System and method for controller-free user drone interaction

ABSTRACT

System and method for controlling an aerial system, without physical interaction with a separate remote device, based on sensed user expressions. User expressions may include thought, voice, facial expressions, and/gestures. User expressions may be sensed by sensors associated with the aerial device or a remote device.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 62/437,386, filed Dec. 21, 2016, the disclosure of which ishereby incorporated by reference in its entirety.

TECHNICAL FIELD

This invention relates generally to the aerial system field, and morespecifically, to a system and method for controller-free aerial systemoperation in the aerial system field.

BACKGROUND OF THE INVENTION

Currently, there are two general approaches in allowing a user tocontrol a drone to take photos and videos. First, the drone may becontroller using a remote controller (RC) or other mobile device, suchas a mobile phone or tablet. In these types of systems, a user mustcontrol the drone manually through physical interaction with the RC ormobile device. This approach provides several shortcomings. First, itrequires hours or days or even months of practice for a user to becomeproficient in controlling the drone. Additionally, not only does theuser have to control operation, i.e., flight, of the drone, but the usermust also control the camera to capture pictures and/or video. Thus, thequality of the image or video is limited by not only the skills ofcontrolling the drone but also the controller's photography orvideography experience.

The second approach is an auto-follow feature. Using the auto-followfeature, the drone or aerial system chooses and locks onto a person andautomatically captures pictures and/or video. Generally, this is theperson that is operating the RC or mobile device, i.e., the “owner”, butmay also be another person, such as a person wearing or being associatedwith a tracking device. This approach also has shortcomings. In general,the movement or drone instructions are relatively simple, i.e., followthe user while capturing pictures and/or video. Using this approach, theresulting pictures and/or video are limited, i.e., always the same or asingle view.

Thus, a more intuitive control approach is desirable. The presentinvention is aimed at one or more of the problems identified above.

SUMMARY OF THE INVENTION

In one aspect of the present invention, an aerial system includes abody, a lift mechanism coupled to the body, an optical system and aprocessing system. The optical system is controllably mounted to thebody by an actuation system. The processing system is coupled to thelift mechanism, the optical system, and the actuation system and isconfigured to:

enter a hover idle state;

recognize a target person;

track target person and scan for user expressions; and,

detect a predefined user expression and responsively execute acorresponding event.

In another aspect of the present invention, method for controlling anaerial system is provided. The method includes the steps of:

entering a hover idle state;

recognizing a target person;

tracking target person and scanning for user expressions; and,

detecting, by a processing system, a predefined user expression andresponsively executing a corresponding event.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic representation of an aerial system and a systemfor controlling the aerial system, according to an embodiment of thepresent invention.

FIG. 2 is a picture of an exemplary aerial system, according to anembodiment of the present invention.

FIG. 3 is a picture of an exemplary optical system, according to anembodiment of the present invention.

FIG. 4 is a second schematic representation of the aerial system,according to an embodiment of the present invention.

FIG. 5 is a third schematic representation of the system for controllingthe aerial system and the aerial system according to an embodiment ofthe present invention.

FIG. 6 is a first flowchart diagram of a method for remote-free usercontrol of an aerial system using user expression, according to anembodiment of the present invention.

FIG. 7 is a second flowchart diagram of a method for remote-free usercontrol of an aerial system using user expression, according to anembodiment of the present invention.

FIG. 8 is a third flowchart diagram of a method for remote-free usercontrol of an aerial system using user expression, according to anembodiment of the present invention.

FIG. 9 is a fourth flowchart diagram of a method for remote-free usercontrol of an aerial system using user expression, according to anembodiment of the present invention.

FIG. 10 is a flowchart diagram of the method for automatic aerial systemoperation.

FIG. 11 is a flowchart diagram of a variation of the method forautomatic aerial system operation.

FIGS. 12 and 13 are a first and second specific example of detecting achange in an orientation sensor signal indicative of an imminentoperation event and automatically operating the lift mechanisms based onthe detected change, respectively.

FIG. 14 is a schematic representation of a first variation of automaticaerial system operation, including a specific example of the detectedsensor signal change indicative of freefall and a specific example oflift mechanism control in response to the detected sensor signal change.

FIG. 15 is a schematic representation of a second variation of automaticaerial system operation including detection of an applied force along asecond axis, further including a specific example of the detected sensorsignal change indicative of freefall and a specific example of liftmechanism control in response to the detected sensor signal change.

FIG. 16 is a schematic representation of a third variation of automaticaerial system operation, in which the aerial system automatically liftsoff a support surface.

FIG. 17 is a schematic representation of a fourth variation of automaticaerial system operation, in which the aerial system automatically hoversupon removal of a support surface.

FIG. 18 is a schematic representation of a first variation of detectinga standby event and operating the aerial system in a standby mode,including a specific example of the detected unexpected sensor signalchange and a specific example of lift mechanism control in response tothe detected standby event.

FIG. 19 is a schematic representation of a second variation of detectinga standby event and operating the aerial system in a standby mode,including detecting an open user hand underneath the aerial system asthe standby event.

FIG. 20 is a schematic representation of a third variation of detectinga standby event and operating the aerial system in a standby mode,including detecting a user hand in a “ready-to-grab” conformation to theside of the aerial system as the standby event.

FIG. 21 is as flowchart diagram of a system for controlling flight of anaerial system based on images presented to the user, according anembodiment of the present invention.

FIG. 22 is a specific example of the control client displaying videorecorded by the aerial system.

FIGS. 23-41 are specific examples of different user input and therespective mapped aerial system actions.

FIG. 42 is a specific example of compensating for aerial systemmovement.

DETAILED DESCRIPTION OF THE INVENTION

The following description of the embodiments of the invention is notintended to limit the invention to these embodiments, but rather toenable any person skilled in the art to make and use this invention.With reference to the drawings and in operation, system 10 forcontrolling an aerial system 12, for example a drone, is provided. Thesystem 10 includes a remote device 14 with a control client 16. Thecontrol client 16 provides a user interface (see below) that allows auser 18 to send instructions to the aerial system 12 to controloperation thereof. As discussed in more depth below, the aerial system12 includes one or more cameras (see below) for obtaining picturesand/or video which may be sent to the remote device 14 and/or stored inmemory on the aerial system 12.

Alternatively, or in addition, the aerial system 12 may include one ormore sensors (see below) for detecting or sensing operations or actions,i.e., expressions, performed by the user 18 to control operation of theaerial system 12 (see below) without direct or physical interaction withthe remote device 14. In controller-free embodiments, the entire controlloop from start (release and hover) to finish (grab and go), as well ascontrolling motion of the aerial system 12 and trigger of events, e.g.,taking pictures and video, are performed solely on board the aerialsystem 12 without involvement of the remote device 14. In some suchembodiments or systems 10, a remote device 14 may not be provided orincluded.

In some embodiments, the remote device 14 includes one or more sensorsthat detect or sense operation or actions performed by the user 18 tocontrol operation of the aerial system 12 without physical interactionwith the remote device 14 under certain conditions, for example, whenthe aerial system 12 is too far from the user 18.

Overview of the System 10 and the Aerial System 12

An exemplary aerial system 12 and control system 10 is shown in FIGS.1-5. The control client 16 of the aerial system 12 functions to receivedata from the aerial system 12, including video images and/or video, andcontrol visual display on the remote device 14. The control client 16may also receive operation instructions and facilitate aerial system 12remote control based on operation instructions. The control client 16 ispreferably configured to execute on a remote device 14, but canalternatively be configured to execute on the aerial system 12 or on anyother suitable system. As discussed above, and more fully below, theaerial system 12 may be controlled solely without direct or physicalinteraction with the remote device 14.

The control client 16 can be a native application (e.g., a mobileapplication), a browser application, an operating system application, orbe any other suitable construct.

The remote device 14 executing the control client 16 functions todisplay the data (e.g., as instructed by the control client 16), receiveuser inputs, compute the operation instructions based on the user inputs(e.g., as instructed by the control client 16), send operationinstructions to the aerial system 12, store control client information(e.g., associated aerial system identifiers, security keys, user accountinformation, user account preferences, etc.), or perform any othersuitable functionality. The remote device 14 can be a user device (e.g.,smartphone, tablet, laptop, etc.), a networked server system, or be anyother suitable remote computing system. The remote device 14 can includeone or more: outputs, inputs, communication systems, sensors, powersources, processing systems (e.g., CPU, memory, etc.), or any othersuitable component. Outputs can include: displays (e.g., LED display,OLED display, LCD, etc.), audio speakers, lights (e.g., LEDs), tactileoutputs (e.g., a tixel system, vibratory motors, etc.), or any othersuitable output. Inputs can include: touchscreens (e.g., capacitive,resistive, etc.), a mouse, a keyboard, a motion sensor, a microphone, abiometric input, a camera, or any other suitable input. Communicationsystems can include wireless connections, such as radios supporting:long-range systems (e.g., Wi-Fi, cellular, WLAN, WiMAX, microwave, IR,radio frequency, etc.), short-range systems (e.g., BLE, BLE long range,NFC, ZigBee, RF, audio, optical, etc.), or any other suitablecommunication system. Sensors can include: orientation sensors (e.g.,accelerometer, gyroscope, etc.), ambient light sensors, temperaturesensors, pressure sensors, optical sensors, acoustic sensors, or anyother suitable sensor. In one variation, the remote device 14 caninclude a display (e.g., a touch-sensitive display including atouchscreen overlaying the display), a set of radios (e.g., Wi-Fi,cellular, BLE, etc.), and a set of orientation sensors.

The aerial system 12 functions to fly within a physical space, capturevideo, stream the video in near-real time to the remote device 14, andoperate based on operation instructions received from the remote device14.

The aerial system 12 can additionally process the video (e.g., videoframes) prior to streaming the video to the remote device 14 and/oraudio received from an onboard audio sensor; generate and automaticallyoperate based on its own operation instructions (e.g., to automaticallyfollow a subject); or perform any other suitable functionality. Theaerial system 12 can additionally function to move the optical sensor'sfield of view within the physical space. For example, the aerial system12 can control macro movements (e.g., large FOV changes, on the order ofmeter adjustments), micro movements (e.g., small FOV changes, on theorder of millimeter or centimeter adjustments), or any other suitablemovement.

As discussed in more detail below, the aerial system 12 can performcertain functionality based on onboard processing of sensor data fromonboard sensors. This functionality may include, but is not limited to:

-   -   Take-off and landing;    -   Owner recognition;    -   Facial recognition;    -   Speech recognition;    -   Facial expression and gesture recognition; and,    -   Control, e.g., motion, of the aerial system based on owner,        facial, expression and gesture recognition, and speech        recognition.

As shown in FIGS. 2-5, the aerial system 12 (e.g., drone) can include abody 20, a processing system 22, a communication system 24, an opticalsystem 26, and an actuation mechanism 28 mounting the optical system 26to the body 20. The aerial system 12 can additionally or alternativelyinclude lift mechanisms, sensors, power system, or any other suitablecomponent (see below).

The body 20 of the aerial system 12 functions to mechanically protectand/or retain the aerial system components. The body 20 can define alumen, be a platform, or have any suitable configuration. The body 20can be enclosed, open (e.g., a truss), or have any suitableconstruction. The body 20 can be made of metal, plastic (e.g., polymer),carbon composite, or any other suitable material. The body 20 can definea longitudinal axis, a lateral axis, a transverse axis, a front end, aback end (e.g., opposing the front end along the longitudinal axis), atop, a bottom (e.g., opposing the top along the transverse axis), or anyother suitable reference. In one variation, while in flight, atransverse axis of the body 20 can be substantially parallel a gravityvector (e.g., perpendicular a ground plane) and the body's longitudinaland lateral axes can be substantially perpendicular the gravity vector(e.g., parallel the ground plane). However, the body 20 can be otherwiseconfigured.

The processing system 22 of the aerial system 12 functions to controlaerial system operation. The processing system 22 can: receive operationinstructions from the communication system 24, interpret the operationinstructions into machine instructions, and control aerial systemcomponents based on the machine instructions (individually or as a set).The processing system 22 can additionally or alternatively process theimages recorded by the camera, stream images to the remote device 14(e.g., in real- or near-real time), or perform any other suitablefunctionality. The processing system 22 can include one or more:processors 30 (e.g., CPU, GPU, etc.), memory (e.g., Flash, RAM, etc.),or any other suitable processing component. In one variation, theprocessing system 22 can additionally include dedicated hardware thatautomatically processes the images (e.g., de-warps the image, filtersthe image, crops the image, etc.) prior to transmission to the remotedevice 14. The processing system 22 is preferably connected to theactive components of the aerial system 12 and mounted to the body 20,but can alternatively be otherwise related to aerial system components.

The communication system 24 of the aerial system functions to sendand/or receive information from the remote device 14. The communicationsystem 24 is preferably connected to the processing system 22, such thatthe communication system 24 sends and/or receives data form theprocessing system 22, but can alternatively be connected to any othersuitable component. The aerial system 12 can include one or morecommunication systems 24 of one or more types. The communication system24 can include wireless connections, such as radios supporting:long-range systems (e.g., Wi-Fi, cellular, WLAN, WiMAX, microwave, IR,radio frequency, etc.), short-range systems (e.g., BLE, BLE long range,NFC, ZigBee, RF, audio, optical, etc.), or any other suitablecommunication system 24. The communication system 24 preferably sharesat least one system protocol (e.g., BLE, RF, etc.) with the remotedevice 14, but can alternatively communicate with the remote device 14via an intermediary communication system (e.g., a protocol translationsystem). However, the communication system 24 can be otherwiseconfigured.

The optical system 26 of the aerial system 12 functions to record imagesof the physical space proximal the aerial system 12. The optical system26 is preferably mounted to the body 20 via the actuation mechanism 28,but can alternatively be statically mounted to the body 20, removablymounted to the body 20, or otherwise mounted to the body 20. The opticalsystem 26 is preferably mounted to the front end of the body 20, but canoptionally be mounted to the bottom (e.g., proximal the front), top,back end, or any other suitable portion of the body 20. The opticalsystem 26 is preferably connected to the processing system 30, but canalternatively be connected to the communication system 24 or to anyother suitable system. The optical system 26 can additionally includededicated image processing hardware that automatically processes imagesrecorded by the camera prior to transmission to the processor or otherendpoint. The aerial system 12 can include one or more optical systems26 of same or different type, mounted to the same or different position.In one variation, the aerial system 12 includes a first optical system26, mounted to the front end of the body 20, and a second optical system26, mounted to the bottom of the body 20. The first optical system 26can actuate about a pivotal support, and the second optical system 26can be substantially statically retained relative to the body 20, withthe respective active surface substantially parallel the body bottom.The first optical sensor 36 can be high-definition, while the secondoptical sensor 36 can be low definition. However, the optical system 26can be otherwise configured.

The optical system 26 can include one or more optical sensors 36 (seeFIG. 5). The one or more optical sensors 36 can include: a single lenscamera (e.g., CCD camera, CMOS camera, etc.), a stereo-camera, ahyperspectral camera, a multispectral camera, or any other suitableimage sensor. However, the optical system 26 can be any other suitableoptical system 26. The optical system 26 can define one or more activesurfaces that receive light, but can alternatively include any othersuitable component. For example, an active surface of a camera can be anactive surface of a camera sensor (e.g., CCD sensor, CMOS sensor, etc.),preferably including a regular array of sensor pixels. The camera sensoror other active surface is preferably substantially planar andrectangular (e.g., having a first sensor edge, a second sensor edgeopposing the first sensor edge, and third and fourth sensor edges eachperpendicular to and extending from the first sensor edge to the secondsensor edge), but can alternatively have any suitable shape and/ortopography. The optical sensor 36 can produce an image frame. The imageframe preferably corresponds with the shape of the active surface (e.g.,rectangular, having a first and second frame edge opposing each other,etc.), more preferably defining a regular array of pixel locations, eachpixel location corresponding to a sensor pixel of the active surfaceand/or pixels of the images sampled by the optical sensor 36, but canalternatively have any suitable shape. The image frame preferablydefines aspects of the images sampled by the optical sensor 36 (e.g.,image dimensions, resolution, pixel size and/or shape, etc.). Theoptical sensor 36 can optionally include a zoom lens, digital zoom,fisheye lens, filter, or any other suitable active or passive opticaladjustment. Application of the optical adjustment can be activelycontrolled by the controller, manually controlled by the user 18 (e.g.,wherein the user manually sets the adjustment), controlled by the remotedevice 14, or otherwise controlled. In one variation, the optical system26 can include a housing enclosing the remainder of the optical systemcomponents, wherein the housing is mounted to the body 20. However, theoptical system 26 can be otherwise configured.

The actuation mechanism 28 of the aerial system 12 functions toactionably mount the optical system 26 to the body 20. The actuationmechanism 28 can additionally function to dampen optical sensorvibration (e.g., mechanically stabilize the resultant image),accommodate for aerial system roll, or perform any other suitablefunctionality. The actuation mechanism 28 can be active (e.g.,controlled by the processing system), passive (e.g., controlled by a setof weights, spring elements, magnetic elements, etc.), or otherwisecontrolled. The actuation mechanism 28 can rotate the optical system 26about one or more axes relative to the body, translate the opticalsystem 26 along one or more axes relative to the body, or otherwiseactuate the optical system 26. The optical sensor(s) 36 can be mountedto the support along a first end, along an optical sensor back (e.g.,opposing the active surface), through the optical sensor body, or alongany other suitable portion of the optical sensor 36.

In one variation, the actuation mechanism 28 can include a motor (notshown) connected to a single pivoted support (e.g., gimbal), wherein themotor pivots the support about the rotational (or gimbal) axis 34 basedon instructions received from the controller. The support is preferablyarranged with the rotational axis substantially parallel the lateralaxis of the body 20, but can alternatively be arranged with therotational axis at any other suitable orientation relative to the body20. The support is preferably arranged within a recessed cavity definedby the body 20, wherein the cavity further encompasses the opticalsensor 36 but can alternatively be arranged along the body exterior orarranged at any other suitable portion of the body 20. The opticalsensor 36 is preferably mounted to the support with the active surfacesubstantially parallel the rotational axis (e.g., with the lateral axis,or axis parallel the lateral axis of the body 20, substantially parallelthe rotational axis), but can alternatively be arranged with the activesurface arranged at any suitable angle to the rotational axis.

The motor is preferably an electric motor, but can alternatively be anyother suitable motor. Examples of electric motors that can be usedinclude: DC motors (e.g., brushed motors), EC motors (e.g., brushlessmotors), induction motor, synchronous motor, magnetic motor, or anyother suitable electric motor. The motor is preferably mounted to thebody 20 (e.g., the body interior), electrically connected to andcontrolled by the processing system 22, and electrically connected toand powered by a power source or system 38. However, the motor can beotherwise connected. The actuation mechanism 28 preferably includes asingle motor-support set, but can alternatively include multiplemotor-support sets, wherein auxiliary motor-support sets can be arrangedorthogonal (or at any other suitable angle to) the first motor-supportset.

In a second variation, the actuation mechanism 28 can include a set ofpivoted supports and weights connected to the optical sensor 36 offsetfrom the optical sensor center of gravity, wherein the actuationmechanism 28 passively stabilizes the optical sensor 36.

A lift mechanism 40 of the aerial system 12 functions to enable aerialsystem flight. The lift mechanism 40 preferably includes a set propellerblades 42 driven by a motor (not shown), but can alternatively includeany other suitable propulsion mechanism. The lift mechanism 40 ispreferably mounted to the body 20 and controlled by the processingsystem 22, but can alternatively be otherwise mounted to the aerialsystem 12 and/or controlled. The aerial system 12 can include multiplelift mechanisms 40. In one example, the aerial system 12 includes fourlift mechanisms 40 (e.g., two pairs of lift mechanisms 40), wherein thelift mechanisms 40 are substantially evenly distributed about theperimeter of the aerial system 12 (e.g., wherein the lift mechanisms 40of each pair oppose each other across the body 20). However, the liftmechanisms 40 can be otherwise configured.

Additional sensors 44 of the aerial system function to record signalsindicative of aerial system operation, the ambient environmentsurrounding the aerial system 12 (e.g., the physical space proximal theaerial system 12), or any other suitable parameter. The sensors 44 arepreferably mounted to the body 20 and controlled by the processingsystem 22, but can alternatively be mounted to any other suitablecomponent and/or otherwise controlled. The aerial system 12 can includeone or more sensors 36, 44. Examples of sensors that can be usedinclude: orientation sensors (e.g., accelerometer, gyroscope, etc.),ambient light sensors, temperature sensors, pressure sensors, opticalsensors, acoustic sensors (e.g., microphones), voltage sensors, currentsensors, or any other suitable sensor.

The power supply 38 of the aerial system 12 functions to power theactive components of the aerial system 12. The power supply 38 ispreferably mounted to the body 20, and electrically connected to allactive components of the aerial system 12 (e.g., directly orindirectly), but can be otherwise arranged. The power supply 38 can be aprimary battery, secondary battery (e.g., rechargeable battery), fuelcell, energy harvester (e.g., solar, wind, etc.), or be any othersuitable power supply. Examples of secondary batteries that can be usedinclude: a lithium chemistry (e.g., lithium ion, lithium ion polymer,etc.), nickel chemistry (e.g., NiCad, NiMH, etc.), or batteries with anyother suitable chemistry.

The methods described herein may be used with one or more aerial systems12, and can optionally be used with a remote computing system, or withany other suitable system. The aerial system 12 functions to fly, andcan additionally function to take photographs, deliver loads, and/orrelay wireless communications. The aerial system 12 is preferably arotorcraft (e.g., quadcopter, helicopter, cyclocopter, etc.), but canalternatively be a fixed-wing aircraft, aerostat, or be any othersuitable aerial system 12. The aerial system 12 can include a liftmechanism 40, a power supply 38, sensors 36, 44, a processing system 22,a communication system 24, a body 20, and/or include any other suitablecomponent.

The lift mechanism 40 of the aerial system functions to provide lift,and preferably includes a set of rotors driven (individually orcollectively) by one or more motors. Each rotor is preferably configuredto rotate about a corresponding rotor axis, define a corresponding rotorplane normal to its rotor axis, and sweep out a swept area on its rotorplane. The motors are preferably configured to provide sufficient powerto the rotors to enable aerial system flight, and are more preferablyoperable in two or more modes, at least one of which includes providingsufficient power for flight and at least one of which includes providingless power than required for flight (e.g., providing zero power,providing 10% of a minimum flight power, etc.). The power provided bythe motors preferably affects the angular velocities at which the rotorsrotate about their rotor axes. During aerial system flight, the set ofrotors are preferably configured to cooperatively or individuallygenerate (e.g., by rotating about their rotor axes) substantially all(e.g., more than 99%, more than 95%, more than 90%, more than 75%) ofthe total aerodynamic force generated by the aerial system 1 (possiblyexcluding a drag force generated by the body 20 such as during flight athigh airspeeds). Alternatively, or additionally, the aerial system 12can include any other suitable flight components that function togenerate forces for aerial system flight, such as jet engines, rocketengines, wings, solar sails, and/or any other suitable force-generatingcomponents.

In one variation, the aerial system 12 includes four rotors, eacharranged at a corner of the aerial system body. The four rotors arepreferably substantially evenly dispersed about the aerial system body,and each rotor plane is preferably substantially parallel (e.g., within10 degrees) a lateral plane of the aerial system body (e.g.,encompassing the longitudinal and lateral axes). The rotors preferablyoccupy a relatively large portion of the entire aerial system 12 (e.g.,90%, 80%, 75%, or majority of the aerial system footprint, or any othersuitable proportion of the aerial system 12). For example, the sum ofthe square of the diameter of each rotor can be greater than a thresholdamount (e.g., 10%, 50%, 75%, 90%, 110%, etc.) of the convex hull of theprojection of the aerial system 12 onto a primary plane of the system(e.g., the lateral plane). However, the rotors can be otherwisearranged.

The power supply 38 of the aerial system functions to power the activecomponents of the aerial system 12 (e.g., lift mechanism's motors, powersupply 38, etc.). The power supply 38 can be mounted to the body 20 andconnected to the active components, or be otherwise arranged. The powersupply 38 can be a rechargeable battery, secondary battery, primarybattery, fuel cell, or be any other suitable power supply.

The sensors 36, 44 of the aerial system function to acquire signalsindicative of the aerial system's ambient environment and/or aerialsystem operation. The sensors 36, 44 are preferably mounted to the body20, but can alternatively be mounted to any other suitable component.The sensors 36, 44 are preferably powered by the power supply 38 andcontrolled by the processor, but can be connected to and interact withany other suitable component. The sensors 36, 44 can include one ormore: cameras (e.g., CCD, CMOS, multispectral, visual range,hyperspectral, stereoscopic, etc.), orientation sensors (e.g., inertialmeasurement sensors, accelerometer, gyroscope, altimeter, magnetometer,etc.), audio sensors (e.g., transducer, microphone, etc.), barometers,light sensors, temperature sensors, current sensor (e.g., Hall effectsensor), air flow meter, voltmeters, touch sensors (e.g., resistive,capacitive, etc.), proximity sensors, force sensors (e.g., strain gaugemeter, load cell), vibration sensors, chemical sensors, sonar sensors,location sensor (e.g., GPS, GNSS, triangulation, etc.), or any othersuitable sensor. In one variation, the aerial system 12 includes a firstcamera mounted (e.g., statically or rotatably) along a first end of theaerial system body with a field of view intersecting the lateral planeof the body; a second camera mounted along the bottom of the aerialsystem body with a field of view substantially parallel the lateralplane; and a set of orientation sensors, such as an altimeter andaccelerometer. However, the system can include any suitable number ofany sensor type.

The processing system 22 of the aerial system functions to controlaerial system operation. The processing system 22 can perform themethod; stabilize the aerial system 12 during flight (e.g., selectivelyoperate the rotors to minimize aerial system wobble in-flight); receive,interpret, and operate the aerial system 12 based on remote controlinstructions; or otherwise control aerial system operation. Theprocessing system 22 is preferably configured to receive and interpretmeasurements sampled by the sensors 36, 44, more preferably by combiningmeasurements sampled by disparate sensors (e.g., combining camera andaccelerometer data). The aerial system 12 can include one or moreprocessing systems, wherein different processors can perform the samefunctionality (e.g., function as a multi-core system), or bespecialized. The processing system 22 can include one or more:processors (e.g., CPU, GPU, microprocessor, etc.), memory (e.g., Flash,RAM, etc.), or any other suitable component. The processing system 22 ispreferably mounted to the body 20, but can alternatively be mounted toany other suitable component. The processing system 22 is preferablypowered by the power supply 38, but can be otherwise powered. Theprocessing system 22 is preferably connected to and controls the sensors36, 44, communication system 24, and lift mechanism 40, but canadditionally or alternatively be connected to and interact with anyother suitable component.

The communication system 24 of the aerial system functions tocommunicate with one or more remote computing systems. The communicationsystem 24 can be a long-range communication module, a short-rangecommunication module, or any other suitable communication module. Thecommunication system 24 can facilitate wired and/or wirelesscommunication. Examples of the communication system 24 include an802.11x, Wi-Fi, Wi-Max, NFC, RFID, Bluetooth, Bluetooth Low Energy,ZigBee, cellular telecommunications (e.g., 2G, 3G, 4G, LTE, etc.), radio(RF), wired connection (e.g., USB), or any other suitable communicationsystem 24 or combination thereof. The communication system 24 ispreferably powered by the power supply 38, but can be otherwise powered.The communication system 24 is preferably connected to the processingsystem 22, but can additionally or alternatively be connected to andinteract with any other suitable component.

The body 20 of the aerial system functions to support the aerial systemcomponents. The body can additionally function to protect the aerialsystem components. The body 20 preferably substantially encapsulates thecommunication system 24, power supply 38, and processing system 22, butcan be otherwise configured. The body 20 can include a platform, ahousing, or have any other suitable configuration. In one variation, thebody 20 includes a main body housing the communication system 24, powersupply 38, and processing system 22, and a first and second frame (e.g.,cage) extending parallel the rotor rotational plane and arranged along afirst and second side of the main body 20. The frames can function as anintermediary component between the rotating rotors and a retentionmechanism (e.g., retention mechanism such as a user's hand). The framecan extend along a single side of the body 20 (e.g., along the bottom ofthe rotors, along the top of the rotors), along a first and second sideof the body 20 (e.g., along the top and bottom of the rotors),encapsulate the rotors (e.g., extend along all sides of the rotors), orbe otherwise configured. The frames can be statically mounted oractuatably mounted to the main body 20.

The frame can include one or more apertures (e.g., airflow apertures)fluidly connecting one or more of the rotors to an ambient environment,which can function to enable the flow of air and/or other suitablefluids between the ambient environment and the rotors (e.g., enablingthe rotors to generate an aerodynamic force that causes the aerialsystem 1 to move throughout the ambient environment). The apertures canbe elongated, or can have comparable length and width. The apertures canbe substantially identical, or can differ from each other. The aperturesare preferably small enough to prevent components of a retentionmechanism (e.g., fingers of a hand) from passing through the apertures.The geometrical transparency (e.g., ratio of open area to total area) ofthe frame near the rotors is preferably large enough to enable aerialsystem flight, more preferably enabling high-performance flightmaneuvering. For example, each aperture can be smaller than a thresholdsize (e.g., smaller than the threshold size in all dimensions, elongatedslots narrower than but significantly longer than the threshold size,etc.). In a specific example, the frame has a geometrical transparencyof 80-90%, and the apertures (e.g., circles, polygons such as regularhexagons, etc.) each of define a circumscribed circle with a diameter of12-16 mm. However, the body can be otherwise configured.

The body 20 (and/or any other suitable aerial system components) candefine a retention region that can be retained by a retention mechanism(e.g., a human hand, an aerial system dock, a claw, etc.). The retentionregion preferably surrounds a portion of one or more of the rotors, morepreferably completely surrounding all of the rotors, thereby preventingany unintentional interaction between the rotors and a retentionmechanism or other object near the aerial system 12. For example, aprojection of the retention region onto an aerial system plane (e.g.,lateral plane, rotor plane, etc.) can overlap (e.g., partially,completely, a majority of, at least 90% of, etc.) a projection of theswept area of one or more of the rotors (e.g., swept area of a rotor,total swept area of the set of rotors, etc.) onto the same aerial systemplane.

The aerial system 12 can additionally include inputs (e.g., microphones,cameras, etc.), outputs (e.g., displays, speakers, light emittingelements, etc.), or any other suitable component.

The remote computing system functions to receive auxiliary user inputs,and can additionally function to automatically generate controlinstructions for and send the control instructions to the aerialsystem(s) 12. Each aerial system 12 can be controlled by one or moreremote computing systems. The remote computing system preferablycontrols the aerial system 12 through a client (e.g., a nativeapplication, browser application, etc.), but can otherwise control theaerial system 12. The remote computing system can be a user device,remote server system, connected appliance, or be any other suitablesystem. Examples of the user device include a tablet, smartphone, mobilephone, laptop, watch, wearable device (e.g., glasses), or any othersuitable user device. The user device can include power storage (e.g., abattery), processing systems (e.g., CPU, GPU, memory, etc.), useroutputs (e.g., display, speaker, vibration mechanism, etc.), user inputs(e.g., a keyboard, touchscreen, microphone, etc.), a location system(e.g., a GPS system), sensors (e.g., optical sensors, such as lightsensors and cameras, orientation sensors, such as accelerometers,gyroscopes, and altimeters, audio sensors, such as microphones, etc.),data communication system (e.g., a Wi-Fi module, BLE, cellular module,etc.), or any other suitable component.

Controller-Free User/Drone Interaction

With reference to FIGS. 1-9, and specifically, to FIGS. 6-9, in oneaspect the present invention provides a system 10 and method forcontroller-free user drone interaction. Normally, the aerial system, ordrone, 12 requires a separate device, e.g., the remote device 14. Theremote device 14 may be embodied in different types of devices,including, but not limited to a ground station, remote control, ormobile phone, etc. . . . In some embodiments, control of the aerialsystem 12 may be accomplished by the user through user expressionwithout utilization of the remote device 14. User expression mayinclude, but is not limited to, any action performed by the user that donot include physical interaction with the remote device 14, includingthought (through brain wave measurement), facial expression (includingeye movement), gesture and/or voice. In such embodiments, userinstructions are received directly via the optical sensors 36 and atleast some of the other sensors 44 and processed by the onboardprocessing system 22 to control the aerial system 12.

In some embodiments, the aerial system 12 may alternatively becontrolled via the remote device 14.

In at least one embodiment, the aerial system 12 may be controlledwithout physical interaction with the remote device 14, however, adisplay of the remote device 14 may be used to display images and/orvideo relayed from the aerial system 12 which may aid the user 18 incontrolling the aerial system 12. In addition, sensors 36, 44 associatedwith the remote device 14, e.g., camera(s) and/or a microphone (notshow) may relay data to the aerial system 12, e.g., when the aerialsystem 12 is too far away from the user 18. The sensor data relayed fromthe remote device 14 to the aerial system 12 is used in the same manneras the sensor data from the on-board sensors 36, 44 are used to controlthe aerial system 12 using user expression.

In this manner, the aerial system 12 may be fully controlled, from startto finish, either (1) without utilization of a remote device 14, or (2)without physical interaction with the remote device 14. In the belowdescribed embodiments, control of the aerial system 12 based on userinstructions received at various on-board sensors 36, 44. It should benoted that in the following discussion, utilization of on-board sensors36, 44 may also include utilization of corresponding or similar sensorson the remote device 14.

In general, the user 18 may utilize certain gestures and/or voicecontrol to control take-off, landing, motion of the aerial system 12during flight and other features, such as triggering of photo and/orvideo capturing. As discussed above, the aerial system 12 may providethe following features without utilization of, or processing by, aremote device 14:

-   -   Take-off and landing;    -   Owner recognition;    -   Facial recognition;    -   Speech recognition;    -   Facial expression and gesture recognition; and,    -   Control, e.g., motion, of the aerial system based on owner,        facial, expression and gesture recognition, and speech        recognition.

In one embodiment, the aerial system 12 may utilize one or more handand/or arm gestures or movements. In general, the hand gestures may berecognized and/or used when the aerial system is a short distance(“close”) to the target and the arm gestures may be recognized and/orused when the aerial system is at a distanced (“distant”) from thetarget. Further, some of the gestures may initiate an action which willcontinue until a secondary event, e.g., recognition of a second gesture(type=“immediate”). Other gestures will initiate an action which willcontinue as long as the gesture is being made. The gestures in thefollowing table are bot exhaustive and are exemplary only:

Range Gesture Instruction Type Close “V” Sign Start Taking PicturesImmediate Close “Ok” Sign Stop Tracking Immediate Close Fist Sign StartTracking Immediate Close Waving Palm Fly Backwards Continuous CloseMoving the Palm Fly in direction of movement Continuous Distant “Y” Pose(arms) Start Taking Pictures Immediate Distant Circle Pose StartOrbiting Immediate Distant Arms Horizontal Stop Orbiting ImmediateDistant Arms Crossed Stop Tracking/Return Immediate Distant Waving ArmFly Forwards Continuous Distant Arm Movement Fly in direction ofmovement Continuous

As detailed above, the aerial system 12 includes an optical system 26that includes one or more optical sensor 36, such as a camera. The atleast one on-board camera is configured for live video streaming andcomputer vision analysis. Optionally the aerial system 12 can have atleast one depth sensor (or stereo-vision pair) for multi-pixel depthsensing. Optionally the aerial system 12 can have at least onemicrophone on board for voice recognition and control.

In general, in order to provide full control of the aerial system 12, aplurality of user/drone interactions or activities from start to end ofan aerial session are provided. The user/drone interactions, include,but are not limited to take-off and landing, owner recognition gesturerecognition, facial expression recognition, and voice control.

In one aspect of the present invention, take-off of the aerial system 12is managed using a release and hover procedure (see below).

After the aerial system 12 is released and hovering, the owner orspecific user must be recognized. In one aspect of the present inventiononly commands or instructions from the owner or specific user arefollowed. In another aspect of the present invention, commands from anyuser within the field of view of the at least one camera may befollowed.

To identify the owned, the aerial system 12, once aloft, mayautomatically spin 360 degrees slowly to search for its owner 18.Alternatively, the aerial system 12 can wait still for the owner 18 toshow up in the field of view. This may be set in the default settings.Once the owner 18 is found, an exemplary default action for the dronesystem 12 is to automatically adjust its own position and orientation toaim the owner at the center of the camera field of view with a preferreddistance (by yawing and/or moving in forward/backward direction). In onepreferred embodiment, after the owner 18 or any person is recognized asthe target, the aerial system 12 can then start tracking the target andscan for gesture commands.

In one aspect of the invention, the owner 18 may be recognized as afunction of visual information, e.g., data received from the opticalsensors 36. For all the commands based on facial expression recognitionand gesture recognition techniques, face recognition is an essentialprerequisite. By pre-registering the face of the user to the aerialsystem 12 (via app or an on-board routine), the aerial system 12 candistinguish the owner from any other persons in the video. Typical facerecognition techniques include principle component analysis (PCA) usingEigen face, elastic bunch graph matching using the Fisherface algorithm,etc.

After the owner's or user's face has been identified, the owner's (orother user's) face may be tracked and the face and its proximal area incaptured images. Commands or instructions from the owner or other user'sfacial expressions and/or gesture can be taken to control the aerialsystem 12. As stated above, however, in other aspect any user in view ofthe camera may control the aerial system 12.

Alternatively, or in addition, voice recognition can be used to identifywhether the voice is from the owner or not. Techniques used to processand store voice prints include frequency estimation, hidden Markovmodels, Gaussian mixture models, pattern matching algorithms, neuralnetworks, matrix representation, Vector Quantization and decision trees.

Gesture recognition may also play an important role in the aerial system12. After the drone is released in the air, gesture input commandbecomes a major user interfacing tool. Gesture recognition can beachieved by using a single RGB camera, and/or a multi-pixel depth sensor(time-of-flight based depth camera, stereovision pair, infrared camerawith structural light pattern, etc.). The state-of-the-art gesturerecognition algorithms can achieve real time recognition with thecomputation run time at 100 ms level on the latest high-end processors.

Recognized gestures, which may be assigned or re-assigned differentfunctions or events, may include, but are not limited to thumb up, thumbdown, open palm, fist, victory gesture, etc. . . . . After a specificgesture is recognized, the assigned event is triggered.

A facial expression recognition implementation contains the followingsteps: raw data pre-processing/detection, feature extraction,classification, post-processing, outputting result. Detection methodscan be categorized as knowledge based, feature based, texture based,skin color based, multiple features, template matching (local binarypatterns LBP), active shape model, appearance based, and distributionfeatures. Typical feature extraction methods include discrete cosinetransform (DCT), Gabor filter, principle component analysis (PCA),independent component analysis (ICA), linear discriminant analysis(LDA). And existing classification methods include Hidden Markov Model(HMM), neural networks, support vector machine (SVM), AdaBoost, etc.

Speech or voice recognition may be used as a tool for command input.Speech or voice recognition techniques may include, but are not limitedto, Hidden Markov model (HMM), Long Short-Term Memory (LSTM) RecurrentNeural Networks (RNN), Time Delay Neural Networks (TDNNs), etc.

With specific reference to FIG. 6, a method M60 for providing userexpression control of the motion of the aerial system 12 according toone embodiment of the present invention is shown. In a first step 60S10,after the aerial system 12 has taken off, the aerial system 12 enters ahover state (see below). In a second step 60S12, the processing system22 searches for and recognizes a target person, e.g., facialrecognition. It should be noted that target person recognition could beaccomplished through other methods, including, but not limited to, useof RFID tags and the like. The target person may be the owner or anyuser with the field of view.

In a third step 60S14 the target (person) is tracked and gesture orexpressions performed by the target are detected and observed. Forexample, visual information and/or audio information, e.g., a picture orvideo of the target's face and an area proximal to the target's face maybe processed to detect predefined user expressions. As stated above,user expressions may include thought, facial expressions, gesturesand/or voice.

In the illustrated embodiment, user gestures performed by the targetshands are used. In a fourth step 60S16, if an open palm is detected,then the method 60 proceeds to a fifth step 60S18. Otherwise, the methodM60 returns to the third step 60S14.

In the fifth step 60S18, the position of the target's palm is trackedrelative to the face of the target. In a sixth step 60S20, if the palmgesture (or open palm of the target) is lost, then the method M60returns to the third step 60S14. In the seventh step 60S22, if movement,i.e., a relative translation of the palm relative to the target, isdetected, then the method M60 proceed to an eighth step 60S24.Otherwise, the method M60 returns to the fifth step 60S28.

In the eighth step 60S24, the aerial system 12 is instructed to move asa function of the relative translation or movement of the open palmdetected in the seventh step 60S22. In a ninth step 60S26, if the palmgesture is lost then the method M60 proceeds to a tenth step 60S28.Otherwise, the method M60 returns to the eighth step 60S24. In the tenthstep 60S28, the aerial system 12 is instructed to stop moving and themethod M60 returns to the third step 60S14.

With reference FIG. 7, a method M70 to initiate an event of the aerialsystem 12 according to an embodiment of the present invention isillustrated. In the illustrated embodiment, the aerial system event maybe triggered in response to detection of a predetermined or pre-defineduser expression. User expression, as discussed above may includethought, facial expression, gesture and/or voice. In a first step 70S12,after the aerial system 12 has taken off and a target user has beenidentified, the aerial system 12 enters a tracking state. In a secondstep 70S14, the target is tracked and the processing system 22 scans forpredetermined/predefined user expressions, e.g., gestures. In a thirdstep 70S16, if a predetermined/predefined expression, e.g., gesture, isdetected, then the method M70 proceeds to a fourth step 70S18.Otherwise, the method 70 returns to the second step 70S14. In the fourthstep 70S18, the event corresponding to the detected predefined orpredetermined expression, e.g., gesture is triggered. Exemplary eventsinclude, but are not limited to taking a picture or snapshot, startshooting video, start (user) auto-follow, and start an auto-capturingroutine.

With reference FIG. 8, a method M80 to terminate a running event of theaerial system 12 according to an embodiment of the present invention isillustrated. In the illustrated embodiment, the aerial system 12 eventmay be terminated in response to detection of a predetermined orpre-defined user expression. User expression, as discussed above mayinclude thought, facial expression, gesture and/or voice. In a firststep 80S12, after the running event has been triggered or initiated (seeabove and FIG. 7). The aerial system 12 enters an event or routinerunning state. In a second step 80S14, the target is tracked and theprocessing system 22 scans for predetermined/predefined userexpressions, e.g., gestures. In a third step 80S16, if apredetermined/predefined termination or touring end expression, e.g.,gesture, is detected, then the method M80 proceeds to a fourth step80S18. Otherwise, the method M80 returns to the second step 80S14. Inthe fourth step 80S18, the event corresponding to the detectedpredefined or predetermined expression, e.g., gesture is terminated.

With reference to FIG. 9, a method M90 to perform an auto-capture eventaccording to an embodiment of the present invention is provided. Themethod M90 allows the user or target to signal to the aerial system 12and trigger an event during which the aerial system 12 moves andpositions itself to a position relative to the target that allows theaerial system 12 to land and/or land safely. For example, the aerialsystem 12 may position itself to a location that allows the user 18 toposition their hand below the aerial system 12, and upon a terminationexpression by the user 18, the aerial system 12 may land and/or allowitself to be captured by the user 18.

In a first step 90S10, the user may release the aerial system 12 and theaerial system 12 takes off and begins to hover. In a second step 90S12,the aerial system 12 enters a hover idle state during which the aerialsystem 12 may begin to search for a target or any user (based on defaultsettings).

In a third step 90S14, the target (or owner) or any user enters a fieldof view of one of the optical system 26 and is recognized. In a fourthstep 90S16, the recognized target (or owner or any user) is tracked andexpressions or gestures of the recognized target are scanned andanalyzed. In a fifth step 90S18, if a user expression corresponding toan auto-capture trigger is detected, then the method M90 proceeds to asixth step 90S20. Otherwise, the method M90 returns to the fourth step90S16.

In the sixth step 90S20, the auto-capturing routing is initiated. In aseventh step 90S22, the processing system 22 automatically controls theaerial system 12 to slowly rotate to look for faces. If in an eighthstep 90S24, a face is found then the method M90 proceeds to a ninth step90S26. Otherwise, the method M90 returns to the seventh step 90S22.

In the ninth step 90S26, the processing system 22 instructs the aerialsystem 12 to adjust its position relative to the target. In a tenth step90S28, if the face of the target is lost, then the method M90 returns tothe seventh step 90S22. Otherwise, the method M90 proceeds to aneleventh step 90S30.

In the eleventh step 90S30, if an expected position (relative to thetarget) is reached, then the method M90 proceeds to a twelfth step90S32. Otherwise, the method M90 returns to the ninth step 90S26.

In the twelfth step 90S32, a picture may be taken. In a thirteenth step90S34, if an auto-capture end or termination expression, e.g., gesture,is detected, then the method M90 proceeds to a fourteenth step 90S36.Otherwise, the method M90 returns to the seventh step 90S22.

In the fourteenth step 90S36, the auto-capture routine is terminated. Ina fifteenth step 90S38, if the aerial system 12 has been retrieved bythe user 18, then the method M90 proceeds to a sixteenth step 90S40.Otherwise, the method M90 returns to the fourth step 90S16. In thesixteenth step 90S40, the aerial system 12 has been grabbed by the user18 and may be shut down.

In one embodiment, of the present invention, before grabbing the droneback, the user 18 may use gesture control/voice control to command thedrone to come closer, making it reachable by the user.

As shown in FIG. 10, a method M100 for automatic aerial system operationmay include: operating the aerial system 12 in a flight mode 100S12,detecting a standby event 100S18, and operating the aerial system 12 ina standby mode 100S20. The method M100 can additionally include:detecting a flight event 100510, receiving a control instruction 100S14,and/or operating the aerial system 12 according to the controlinstruction 100S16.

The method functions to automatically cease aerial system flight,independent of control instruction receipt. In a first variation, theaerial system automatically detects that the aerial system 12 has beenrestrained during flight and automatically operates in a standby mode inresponse to determination of aerial system restraint. In a specificexample, the aerial system 12 slows down or stops the lift mechanism 40once it detects that a user has grabbed the aerial system mid-flight ormid-air (e.g., as shown in FIG. 11). In a second variation, the aerialsystem automatically identifies a landing site and automaticallyoperates to land on the landing site. In a first specific example, theaerial system automatically detects a user's hand below the aerialsystem 12 (e.g., using a camera with a field of view directed downwardand visual analysis methods) and gradually slows the propeller speed toland the aerial system 12 on the user's hand. In a second specificexample, the aerial system automatically detects a landing site in frontof the aerial system 12, automatically flies toward the landing site,and automatically controls the lift mechanism 40 to land on the landingsite. However, the method can otherwise cease aerial system flight.

The method can additionally function to automatically fly the aerialsystem 12, independent of control instruction receipt. In a firstvariation, the aerial system automatically hovers (e.g., in place) whenthe aerial system 12 is released (e.g., from a user's hand). In a secondvariation, the aerial system automatically flies along a forceapplication vector, stops, and hovers in response to the aerial system12 being thrown or pushed along the force application vector. In a thirdvariation, the aerial system 12 can automatically take off from a user'shand. However, the method can otherwise fly the aerial system 12.

This method can confer several benefits over conventional systems.First, by automatically entering an aerial system standby mode,automatically flying in response to aerial system release, and/orautomatically landing on a user's hand or user-specified landing site,the method enables more intuitive user interactions with the aerialsystem 12. Second, by automatically operating independent of outsidecontrol instruction receipt, the method frees a user from controllingthose aspects of aerial system flight. This can enable the user tocontrol auxiliary systems (e.g., camera systems), minimize multitasking,or otherwise reduce user interaction required for aerial system flight.However, the method can confer any other suitable set of benefits.

Detecting a flight event 100S10 functions to detect an imminentoperation event 110S10 requiring or otherwise associated with aerialsystem flight. The imminent operation event 110S10 can be freefall(e.g., aerial system motion along a first axis parallel a gravityvector), imminent freefall, aerial system arrangement in a predeterminedorientation (e.g., arrangement with a major aerial system plane within apredetermined range from perpendicular to a gravity vector for apredetermined amount of time, such as 0.5 s), manual support of theaerial system 12 in mid-air (e.g., based on the acceleration patterns,rotation patterns, vibration patterns, temperature patterns, etc.), orbe any other suitable imminent operation event. 100S10 preferablyincludes detecting a change in a sensor signal associated with imminentoperation. The change is preferably detected by the processing system 22based on signals received from the on-board sensors 36, 44 (e.g.,orientation sensors), but can alternatively be detected by the remotecomputing system (e.g., wherein the sensor signals are transmitted tothe remote computing system), or detected by any other suitable system.The predetermined change can be set by a manufacturer, received from theclient running on the remote computing system, received from a user 18,or otherwise determined. The change can be determined: at apredetermined frequency, every time a new orientation sensor signal isreceived, or at any other suitable time. The predetermined change can bea signal change, a parameter change (e.g., amount of accelerationchange, velocity change, etc.), a rate of change (e.g., rate ofacceleration change), or be any other suitable change.

The change indicative of imminent operation can be received from a user18, received from the client, automatically learned (e.g., based on atrained learning set of labeled accelerometer patterns), or otherwisedetermined. The actual change can be considered a change indicative ofimminent operation if the actual change substantially matches thepredetermined change indicative of imminent operation, is classified asa change indicative of imminent operation, substantially matches apattern parameter values indicative of imminent operation, or can beotherwise detected.

The orientation sensor signals can be periodically monitored for thepredetermined change, wherein monitoring the signals can includetemporarily caching a set of prior orientation sensor signals,determining a change between the cached orientation sensor signals and anew orientation sensor signal. However, the orientation sensor signalscan be otherwise monitored. In one embodiment (shown in FIG. 12), thepredetermined change can be the acceleration (e.g., proper acceleration)or a component of the acceleration (e.g., along an axis associated witha gravity vector) becoming substantially equal to zero (e.g., less than0.1 g, less than 0.3 g, less than a threshold fraction of a typicalacceleration observed in the aerial system 12, such as 10% or 30%,etc.), dropping toward zero, dropping toward zero beyond a thresholdrate, or exhibiting any other suitable absolute change, pattern ofchange, or other change indicative of freefall. The axis associated witha gravity vector can be an axis parallel the gravity vector, apredetermined aerial system axis and/or orientation sensor axis (e.g., acentral axis perpendicular to a lateral plane of the aerial system), orbe any other suitable axis. In a specific example, detecting the flightevent 100S10 includes detecting a proper acceleration substantiallyequal to zero at an accelerometer mounted to the aerial system body.

In a first variation of this embodiment, the axis can be the axisperpendicular the bottom of the aerial system 12 (e.g., bottom of theaerial system housing). In a second variation, the aerial system 12 canautomatically identify the axis parallel the gravity vector. This caninclude identifying the axis for which a measured acceleration that issubstantially the same as or higher than the magnitude of gravityacceleration was measured (e.g., for a predetermined period of time). Inthis variation, upon determination that the predetermined change hasoccurred, the method can additionally include analyzing the sensormeasurements from other axes to determine whether the aerial system 12is truly in freefall (e.g., wherein the measurements from other axes areless than the gravity acceleration magnitude) or has simply been rotated(e.g., wherein the measurements from one or more other axes is more thanor equal to the gravity acceleration magnitude). Additionally, oralternatively, in this variation, the method can include correlating theacceleration measurements with disparate orientation information (e.g.,measurements from one or more sensors such as a gyroscope or camera).The method can optionally selectively ignore or not considermeasurements for certain axes (e.g., longitudinal axis of the aerialsystem 12). However, the axis can be otherwise determined, or no singleaxis may be used (e.g., instead relying on a total magnitude).

In a second embodiment (shown in FIG. 13), altimeter signals can beperiodically monitored for a predetermined change. The predeterminedchange can be a predetermined decrease in altitude, a predetermined rateof altitude change, or be any other suitable change.

In a third embodiment, accelerometer and/or gyroscope signals can beperiodically monitored for an indication that the aerial system 12 isbeing supported in a substantially horizontal orientation (e.g., an axisperpendicular the bottom of the aerial system 12 is within a thresholdangle from a gravity vector, such as 1°, 5°, 10°, or 15°). In oneexample, the flight event is detected 100510 when the spatial sensorsignals indicate that the aerial system 12 has been supportedsubstantially horizontally for greater than a threshold time (e.g., 100ms, 350 ms, 1 s, 2 s, 5 s, etc.) while the aerial system 12 is in astandby state and the sonar and optical sensors are sampling valid datafor flight control. However, the change indicative of imminent operationcan be otherwise determined.

Operating the aerial system 12 in a flight mode 100S12 functions to flythe aerial system 12. 100S12 preferably includes operating the liftmechanism 40 in a flight mode, but can additionally or alternativelyinclude operating any other suitable aerial system components in aflight mode. The aerial system 12 is preferably automatically operatedby the processing system 22, but can alternatively be automaticallyoperated by the remote computing system or by any other suitable system.The aerial system 12 is preferably operated in a flight mode 100S12automatically in response to detecting the flight event 100510, but canadditionally or alternatively be operated after a predetermined timeduration has passed after the flight event is detected 100510, after theaerial system altitude has changed beyond a predetermined altitudechange (e.g., as determined from the altimeter), or at any othersuitable time. The aerial system 12 is preferably operated according toa set of operation parameters, wherein the operation parameters can bepredetermined, selected (e.g., based on the sensor measurementcombination at the time of or preceding change detection; based on theclassification of the sensor measurement patterns or combination; etc.),or otherwise determined. The operation parameters can include: powerprovided to the lift mechanism 40 (e.g., voltage, current, etc.), liftmechanism 40 speed or output, timing, target sensor measurements, or anyother suitable operation parameter.

The aerial system 12 can operate in the flight mode using signals from:a front-facing camera, a downward-facing camera, orientation sensors, alaser system (e.g., rangefinder, LIDAR), radar, stereo-camera system,time of flight, or any other suitable optical, acoustic, range finding,or other system. The aerial system 12 can process the signals using RRT,SLAM, kinematics, optical flow, machine learning, rule-based algorithms,or any other suitable method. In a specific example, the path movementmode includes sampling a series of images with a front-facing camera andautomatically determining the aerial system physical position within a3-D space using the series of images and a location method (e.g., SLAM)running on-board the aerial system 12. In a second specific example, thepath movement mode includes sampling a series of images with adown-facing camera (e.g., sampling at 60 fps, or at any other suitablefrequency), automatically detecting apparent movement between the aerialsystem 12 and the ground based on the sampled images (e.g., usingoptical flow), which can assist in determining aerial system position orkinematics (e.g., speed, acceleration), and automatically correcting theaerial system balance or position based on the detected apparentmovement. In a third specific example, the aerial system locationdetermined using the first specific example and the aerial systemkinematics determined using the second specific example can be fed intoa flight control algorithm to hover, fly, or otherwise control theaerial system 12.

The flight mode preferably includes a hover mode in which the aerialsystem position in the air (e.g., vertical position, lateral position,etc.) is substantially maintained, but can alternatively be any othersuitable flight mode. The flight mode preferably includes maintaining anaerial system orientation such that a central axis normal to a lateralplane of the aerial system 12 is substantially parallel to a gravityvector (e.g., within 20°, within 10°, within 3°, within 1°, etc.).However, the central axis can be otherwise maintained. The flight modepreferably includes generating a force at the lift mechanism 40 equaland opposite the force exerted on the aerial system 12 by gravity (e.g.,to hover), but can alternatively include generating a vertical forcegreater or lesser than the gravitational force (e.g., to increase ordecrease altitude, and/or to arrest vertical movement and bring theaerial system 12 into a hovering state). The flight mode canadditionally or alternatively include generating a non-vertical forceand/or a torque (e.g., to change the aerial system pitch or roll, tocause or arrest lateral movement, etc.). For example, the flight modecan include detecting an orientation, position, and/or velocity change,determining that the change is due to wind and/or another externalperturbation such as a collision (e.g., classifying the change as a windand/or collision event, determining a probability of wind perturbation,determining a probability of the perturbation being a grab event, etc.),and operating the lift mechanism 40 to correct for the change and returnto an original or desired position, orientation, and/or velocity.

The flight mode can additionally or alternatively include a pathmovement mode (e.g., flying in a straight line, flying along apredetermined path, etc.), a program mode (e.g., flying along a pathdetermined dynamically based on a flight program, flying based on facialand/or body tracking such as following or orbiting around a person ormaintaining the person's face within a camera field of view, etc.),and/or any other suitable mode. The flight mode can optionally includecapturing an image (e.g., storing a single image, streaming a video,etc.) using an aerial system camera mounted (or otherwise mechanicallycoupled) to the body 20.

The flight mode can additionally or alternatively include an imagingmode, wherein the aerial system automatically identifies an imagingtarget (e.g., person, face, object, etc.) and controls its flight toautomatically follow the imaging target through a physical space. In onevariation, the aerial system 12 can run object recognition and/ortracking methods, facial recognition and/or tracking methods, bodyrecognition and/or tracking methods, and/or any other suitable method onthe sampled images (e.g., from the front-facing camera) to identify andtrack the imaging target. In a specific example, the aerial system 12can automatically image a substantially 360° region about itself (e.g.,by rotating about the central axis, by moving the camera around, byusing a 360° camera, etc.), automatically identify imaging targets fromthe image, and automatically follow an imaging target (e.g.,automatically identified or manually selected) about the physical space.However, the imaging mode can be otherwise performed. However, theflight mode can include any other suitable set of operation modes.

The aerial system 12 can be operated in the flight mode by independentlycontrolling the angular velocity of each rotor and/or the powerdelivered to each rotor. However, the rotors can be controlled as agroup or in any other suitable manner. 100S12 preferably includesgenerating an aerodynamic force at the set of rotors that issubstantially equal to the total aerodynamic force generated by theaerial system 12, more preferably also substantially equal to the netforce exerted by the aerial system 12 (e.g., wherein the aerial system12 does not include any other components configured to generatesignificant aerodynamic force, or to otherwise exert significant force,such as propulsive force, on the ambient environment).

In one variation, operating the aerial system 12 in the flight mode caninclude spooling up the rotor angular velocity of each rotor to a flightrotor speed (e.g., at which the set of rotors generates a flightaerodynamic force) from a standby rotor speed (e.g., at which the set ofrotors generates a standby aerodynamic force lower than the flightaerodynamic force, such as substantially zero force or a small fractionof the flight aerodynamic force). In this variation, the flight rotorspeed is preferably the hover rotor speed at which the aerial systemhovers; alternatively, the speed can be any other suitable rotationspeed. The flight speed can be preset (e.g., by a manufacturer),received from the client, automatically determined (e.g., based on therate of signal change), or otherwise determined. The standby rotor speedcan be low speed (e.g., a proportion of the hover speed), substantiallyzero angular velocity (e.g., wherein the rotors are not rotating), orhave any other suitable speed. The standby rotor speed can be preset(e.g., by a manufacturer), received from the client, or otherwisedetermined. The rotor speed can be immediately transitioned from thestandby rotor speed to the flight rotor speed, transitioned based on therate of orientation sensor signal change, transitioned at apredetermined rate, or transitioned in any other suitable manner.

In a first example, the rotation speed is first increased to a speedabove the hover speed, then lowered to the hover speed, such that theaerial system ceases freefall and hovers after freefall is detected.This can function to prevent the aerial system 12 from freefalling whena support surface is suddenly removed (shown in FIG. 17). In a secondexample, the rotation speed can be proportionally related to the rate ofacceleration change. In a specific example, the rotation speed can befaster than the hover speed when the acceleration change exceeds thatassociated with freefall (e.g., when the aerial system is thrown down).This can function to enable the aerial system 12 to recover fasterand/or recover an initial altitude (e.g., measured before or when thechange was detected). In a second specific example, the rotation speedcan be increased proportionally to the amount of acceleration change. Inoperation, this causes the rotors to gradually spool up as the aerialsystem 12 is gradually released by the user 18 (shown in FIG. 14). In athird specific example, the rotor speed can increase at a predeterminedrate. In operation, this causes the rotors to gradually spool up, slowlylifting the aerial system away from a support surface, such as a user'shand (shown in FIG. 16). In this specific example, the method canadditionally include switching to the first example when the supportsurface is suddenly removed (e.g., as determined from a sudden change inthe orientation sensor signal). The rotation speed can optionally belimited to prevent or minimize wake effects. However, the lift mechanismcan be otherwise operated in response to detection of the change.

The method can optionally include monitoring sensor signals associatedwith a second axis and determining the lift mechanism operationparameters (for lift mechanism operation in response to imminentoperation detection) based on the sensor signals for the second axis.This can function to select lift mechanism operation parameters thatenable the aerial system traverses a distance along the second axisbefore halting and hovering. The second axis is preferably differentfrom the axis substantially parallel to the gravity vector (e.g.,perpendicular the axis substantially parallel to the gravity vector, ata non-zero angle to the axis, etc.), but can alternatively be the same.The axes can be fixed with respect to the aerial system 12, or can bedynamically transformed (e.g., to attempt to fix the axes with respectto gravity and/or the ambient environment, possibly based onmeasurements sampled by the accelerometer, gyroscope, camera, and/or anyother suitable sensors). The sensor signals for the second axis that areconsidered in determining the lift mechanism operation parameters can besensor signals acquired concurrently with the sensor signals for thefirst axis, before the imminent operation change is detected, after theimminent operation change is detected (e.g., in response to changedetection), or at any other suitable time. The distance can bepredetermined, determined based on time (e.g., the aerial system 12 cantraverse along the second axis for is after release), determined basedon the amount of applied force, or be determined in any other suitablemanner.

In one variation, shown in FIG. 15, the second axis can be parallel thelongitudinal axis of the body (e.g., intersect the camera field ofview). In response to detecting force application along the second axis(e.g., within a time window of change detection), the aerial system 12can automatically determine lift mechanism operation instructions tocounteract the applied force. This can function to allow the aerialsystem 12 to travel a predetermined distance along the second axisbefore ceasing further traversal. Force application and/or applied forcemagnitude can be determined from an orientation sensor monitoring thesecond axis (e.g., the accelerometer for the second axis), determinedfrom a force sensor arranged along an aerial system surfaceperpendicular the second axis, or otherwise determined. The appliedforce to be counteracted can be the instantaneous force in the secondaxis at the time the predetermined condition is met, the applied forcemeasured within a time window of imminent operation event detection(e.g., the maximum force, minimum amount of force, etc.), the appliedforce measured concurrent with imminent operation event detection, or beany other suitable force measured at any other suitable time. In oneexample, the lift mechanism operation instructions can include spoolingup the rotors to hover the aerial system immediately after imminentoperation event detection, allowing the aerial system 12 to coast usingthe applied force for a predetermined period of time after imminentoperation event detection, controlling the lift mechanisms to ceasefurther traversal along the second axis (or any axis) after apredetermined condition has been met, and controlling the liftmechanisms 40 to hover the aerial system 12 (e.g., controlling the liftmechanisms 40 to operate at hover speed). In a second example, the liftmechanism operation instructions can include determining the resultantaerial system speed or acceleration along the second axis due to theapplied force, spooling up the rotors to maintain the aerial systemspeed or acceleration along the second axis immediately after imminentoperation event detection until a predetermined condition has been met,controlling the lift mechanisms 40 to cease further traversal along thesecond axis (or any axis) upon satisfaction of the predeterminedcondition, and controlling the lift mechanisms 40 to hover the aerialsystem 12 (e.g., controlling the lift mechanisms 40 to operate at hoverspeed). The predetermined condition can be imminent operation eventdetection (e.g., wherein the instructions are implemented immediatelyafter imminent operation event detection), within a threshold period oftime after imminent operation event detection, after a predeterminedcondition is met after imminent operation event detection (e.g., after apredetermined distance has been traversed, after a predetermined amountof time has passed, etc.), or at any other suitable time. In oneexample, the predetermined condition can be selected based on themagnitude of applied force (e.g., acceleration magnitude, etc.). Themagnitude of the applied force can be the magnitude of the force appliedalong the second axis, the total magnitude of the force applied to thesystem (e.g., less the force applied by gravity), or be otherwisedetermined.

In a first specific example, the instruction execution delay can beproportional to the amount of applied force, such that the aerial systemflies further before halting further aerial system traversal along thesecond axis when larger forces are applied upon aerial system release.In a second specific example, the instruction execution delay can beinversely proportional to the amount of applied force, such that theaerial system flies a shorter distance before halting when larger forcesare applied upon aerial system release. However, the aerial system 12can be otherwise operated based on sensor signals for the second axis.

The method can optionally include monitoring sensor signals associatedwith aerial system altitude and determining the lift mechanism operationparameters based on the altitude. In one variation, this can function toselect lift mechanism operation parameters to regain an initial aerialsystem altitude (e.g., compensate for any altitude losses due tofreefall prior to recovery). The altitude can be determined based onsignals sampled by an altimeter, and/or a relative altitude can bedetermined based on image analysis, range finding (e.g., using avertically-oriented rangefinder to determine distance to the ground,floor, and/or ceiling). The altimeter signals (and/or other altitudedata) that are considered in determining the lift mechanism operationparameters can be altimeter signals acquired concurrently with thesensor signals for the first axis, before the imminent operation changeis detected, after the imminent operation change is detected (e.g., inresponse to change detection), or at any other suitable time. Forexample, the method can include determining the initial aerial systemaltitude within a predetermined time window from imminent operationevent detection (e.g., prior to imminent operation event detection,based on altimeter measurements recorded prior to imminent operationevent detection), spooling up the rotors to hover the aerial systemimmediately after imminent operation event detection, and increasing therotor speed until the aerial system reaches the initial aerial systemaltitude after the aerial system 12 is stabilized. However, thealtimeter signals (and/or other altitude data) can be used in any othersuitable manner.

Receiving a control instruction 100S14 can function to enable a user 18to augment and/or override automatic aerial system operation. Thecontrol instruction is preferably received during aerial system flight,but can additionally or alternatively be received before flight and/orat any other suitable time. The processing system 22 preferably receivesthe control instructions, but any other suitable system can receive thecontrol instructions. The control instructions are preferably receivedfrom a user, user device, remote controller, and/or client (e.g.,running on a user device) associated with the aerial system 12, but canalternatively be received from a location associated with the aerialsystem 12 (e.g., from a device at the location), sensors on-board theaerial system 12 (e.g., interpreting hand or body signals), and/or fromany other suitable system. The user 18 can be recognized by the aerialsystem 12 (e.g., through optical recognition, such as facial or bodyrecognition), can be near the aerial system 12 (e.g., within range ofthe aerial system sensors), can be otherwise associated with the aerialsystem 12, or can be any suitable user 18. The user device and/or clientcan be paired with the aerial system 12 (e.g., through a Bluetoothconnection, dynamically paired upon aerial system startup, paired at themanufacturing facility, etc.), have a complementary security key pairfor the aerial system 12, be associated with the same user account asthe aerial system 12, or be otherwise associated with the aerial system12. The control instructions can be generated by the user 18, generatedby the user device or client (e.g., in response to user input receipt),generated by a device at the location associated with the aerial system12, determined based on a characteristic of the control instructionsender (e.g., location appearance characteristic, ambient environmentaudio characteristic, etc.), generated by the aerial system 12, and/orgenerated or determined in any other suitable manner.

In one variation, the control instructions can include a landinginstruction. In a first embodiment, 100S14 includes determining alanding area (e.g., automatically identifying the landing area). Thiscan be performed wholly or partially by the processing system 22, remotecomputing system, or any other suitable system. The landing area can beautomatically determined based on aerial system sensor measurements, bereceived from the control instruction sender, be user-specified (e.g.,at the client), or be otherwise determined.

In a first variation of this embodiment, a retention mechanism (e.g.,human hand, docking station, capture device, etc.) is determined to bethe landing area, based on the position, type, and/or conformation ofthe retention mechanism. This variation preferably includes opticallydetecting (e.g., using image recognition techniques, classificationtechniques, regression techniques, rule-based techniques,pattern-matching techniques, etc.) the retention mechanism position,type, and/or conformation, but can additionally or alternatively includedetermining the position, type, and/or conformation in any othersuitable manner.

For example, the retention mechanism can be a human hand. In a firstspecific example, the landing area is an open hand detected using imagesfrom the downward facing camera (e.g., as shown in FIG. 19). In a secondspecific example, the landing area is a hand in a “ready-to-grab”conformation (e.g., as shown in FIG. 20). In a third specific example,the landing area is a hand making a beckoning gesture.

This variation can include: periodically analyzing (e.g., using visualanalysis techniques, image analysis techniques, etc.) sensor data suchas images captured by aerial system cameras (e.g., arranged along thetop, side, and/or bottom of the aerial system) for a retention mechanismin a predetermined conformation type (e.g., open hand, “ready-to-grab”hand, etc.), and identifying the retention mechanism as a landing areain response to detection of parameters indicative of the predeterminedconformation type.

In a first example, the method can include: sampling a set of infraredimages, identifying a region within the image having an infraredsignature above a threshold value, and determining that the identifiedregion is a hand (e.g., using pattern matching, deterministic methods,classification, regression, probabilistics, etc.). For example, theidentified region can be determined to be a hand when the regionperimeter substantially matches a reference pattern for a hand. In asecond example, the method can include: sampling a set of visual rangeimages, segmenting the image background from the foreground, anddetermining that the foreground region is a hand (e.g., using methodsdiscussed above). However, a human hand can be otherwise identified.

This variation can optionally include: identifying a user hand fromimages (e.g., recorded by a downward facing camera) and identifying thehand as a landing area in response to recognizing the hand as a specificuser's hand (e.g., using classification techniques, regressiontechniques, biometric data such as fingerprints, etc.) associated withthe aerial system 12. For example, extracted biometric data can becompared with biometrics can be stored on the aerial system 12, in theuser device, or in a remote database, wherein the user can be rejectedif the biometric data does not match beyond a threshold percentage, andaccepted if the biometric data matches beyond the threshold percentage.This embodiment can optionally include ignoring commands received from ahand (e.g., identifying the hand as a non-landing area) when thedetected hand is not associated with a user associated with the aerialsystem.

In a second variation, the landing area can be a substantially flatsurface (e.g., perpendicular a gravity vector) proximal the aerialsystem 12 (e.g., identified based on based on visual and/or imageprocessing of images recorded by a front facing or downward facingcamera, identified by a beacon near the landing area, specified by aninstruction received from a user device, etc.). In a third variation,the landing area can be a predetermined docking area (e.g., a home base,identified by an optical pattern, beacon signal, predeterminedgeographic location, or otherwise identified). However, the landing areacan be any other suitable landing area and/or can be otherwisedetermined.

In a second embodiment, the landing instruction includes a time and/ortime period. For example, the landing instruction can include a time toland, a desired flight duration (e.g., measured from the flight eventdetection time, from the stabilization time, from the landinginstruction receipt time, etc.), and/or any other suitable timinginformation.

Additionally, or alternatively, the control instruction can include aflight instruction (e.g., speed, altitude, heading, flight pattern,target destination, collision avoidance criteria, etc.), a sensorinstruction (e.g., begin video streaming, zoom camera, etc.), and/or anyother suitable instruction.

Operating the aerial system 12 according to the control instruction100S16 functions to carry out the control instruction. 100S16 ispreferably performed automatically in response to receiving the controlinstruction 100S14, but can additionally or alternatively be performedat any suitable time after receiving the control instruction 100S14. Theprocessing system 22 preferably operates the lift mechanisms 40 and/orother aerial system modules based on the control instructions, butadditionally or alternatively, any other suitable system can operate theaerial system 12. In a first variation, the control instructionsoverride the automatic flight instructions. In a second variation, thecontrol instructions are augmented by the automatic flight instructions(e.g., wherein the processor generates a tertiary set of flightinstructions based on the automatic flight instructions determined basedon the sensor data and the received control instructions). In a thirdvariation, the control instructions are executed after a predeterminedflight state has been reached. In one example of the third variation,the control instructions are executed after the aerial system 12 hasstabilized (e.g., has substantially ceased traversal and/or ishovering). However, the control instructions can be executed at anysuitable time in any suitable manner. After performing 100S16, theaerial system 12 can resume operating in a previous mode (e.g., theoperation mode immediately before performing 100S16, such as a hovermode), can begin operation in a different flight mode, can enter astandby mode, and/or can operate in any other suitable mode.

In a first embodiment, the control instruction includes a flightinstruction, and 100S16 can include operating according to the flightinstruction. For example, in response to receiving a command to increasealtitude and pan left, 100S16 can include automatically operating thelift mechanism 40 to follow the instructions, and then to resume aerialsystem hovering in the new position. In a second example, in response toreceiving a command to increase rotor speed, 100S16 can includeincreasing the rotor speed accordingly.

In a second embodiment, the control instruction is a landing instructionincluding a landing area, and 100S16 can include automaticallygenerating a flight path to the landing area, generating lift mechanismoperation instructions to follow the generated flight path, andexecuting the instructions. This can function to automatically land thelift mechanism 40. The flight path can be generated based on theintervening physical volume between the aerial system 12 and the landingarea (e.g., as determined based on visual and/or image processing ofimages recorded by a front facing or downward facing camera), be apredetermined flight path, or otherwise determined. In one example,determining the flight path and/or lift mechanism operation instructionsincludes: determining the distance between the aerial system and thelanding area (e.g., based on LIDAR, the relative size of a referenceobject or point within the field of view, etc.), and determining a rotorspool down rate based on the instantaneous rotor speed, the standbyrotor speed, and the distance. In a second example, determining theflight path and/or lift mechanism operation instructions includestracking the landing area (e.g., to track flight progress toward thelanding area, to track the current position of a moving landing area,etc.) and automatically controlling the aerial system 12 to land on thelanding area. However, the lift mechanism operation instructions can beotherwise generated.

In a first specific example, in which the landing area is an open hand,100S16 includes automatically controlling the aerial system 12 to landon the open hand (e.g., operating the lift mechanism 40, such as byreducing the rotor speeds, to slowly lower the aerial system 12 onto theopen hand) in response to detecting the open hand. In a second specificexample, in which the landing area is a “ready-to-grab” hand, 100S16includes automatically controlling the aerial system 12 to fly proximalthe hand (e.g., within reach of the hand, in contact with the hand,within a threshold distance of the hand, such as 1 in, 3 in, or 1 foot,etc.) in response to detecting the hand (e.g., immediately afterdetecting the hand, a period of time after detecting the hand, beforedetecting a standby event 100S18 and/or operating in a standby mode100S20, etc.). However, the aerial system 12 can be operated accordingto the control instruction 100S16 in any suitable manner.

Detecting a standby event 100S18 functions to indicate that the aerialsystem 12 should commence a standby procedure. The standby event (e.g.,flight cessation event) is preferably detected while the aerial system12 is operating in a flight mode (e.g., hover mode, landing mode, etc.),but can additionally or alternatively be detected while the aerialsystem 12 is operating in any other suitable mode and/or at any othersuitable time. The standby event is preferably detected by theprocessing system 22 (e.g., of the aerial system), but can alternativelybe automatically detected by the remote computing system, the userdevice, or by any other suitable system.

Detecting the standby event 100S18 preferably includes detecting a grabindication (e.g., indication that the aerial system has been captured orseized by a retention mechanism such as a human hand) and/or holdingindication (e.g., indication that the aerial system 12 is in prolongedcontact with a user 18, indication that the aerial system 12 is dockedat a docking station, etc.), and can additionally or alternativelyinclude detecting a landing indication (e.g., indication that the aerialsystem has landed on and/or is supported by a landing area), proximityindication (e.g., user proximity, landing area proximity, etc.), and/orany other suitable standby indication. The standby event is preferablydetected based on data sampled by sensors, more preferably on-boardaerial system sensors (e.g., inertial measurement unit, camera,altimeter, GPS, temperature sensor, etc.). For example, the standbyevent can be detected based on the value and/or change in value of theaerial system's: orientation (e.g., orientation with respect to gravity,orientation change and/or rate of change, etc.), altitude (e.g.,altitude change and/or rate of change; determined based on altimeterreadings, image processing, etc.), temperature (e.g., increasing aerialsystem temperature, temperature difference between regions of the aerialsystem 12, etc.), and/or force (e.g., aerial system compression).However, the standby event can additionally or alternatively be detectedbased on transmissions (e.g., from a remote control such as a client ofa user device) and/or any other suitable information.

The standby event can be detected using classification, regression,pattern matching, heuristics, neural networks, and/or any other suitabletechniques. Monitoring and analyzing data to detect the standby eventpreferably includes discriminating between standby events (e.g., grabevents, etc.) and other events (e.g., wind events, collision events,etc.). For example, the method can include monitoring aerial systemsensor data while operating in a flight mode, detecting a firstanomalous event and classifying it as a wind event (e.g., flightdisturbance due to wind), then detecting a second anomalous event andclassifying it as a grab event.

In a first variation, detecting the standby event 100S18 includesdetecting an unexpected spatial sensor signal change. The unexpectedspatial sensor signal change can be indicative of a user grabbing theaerial system mid-flight or mid-air, or be indicative of any othersuitable event. The unexpected spatial sensor signal change can be achange relative to another spatial sensor signal (e.g., previous signalfrom the spatial sensor, previous or concurrent signal from a differentspatial sensor, etc.), change relative to an expected spatial sensorsignal (e.g., corresponding to a target or desired aerial systemorientation, velocity, and/or other spatial parameter, based on liftmechanism control, etc.), and/or any other suitable spatial sensorsignal change. In a first embodiment of this variation, detecting anunexpected spatial sensor signal change includes detecting a spatialsensor signal change (e.g., gyroscope signal change, accelerometerchange, IMU change, altimeter change, etc.) different from expectedspatial sensor signals determined based on automatically generatedand/or remotely received control instructions. In a first example ofthis embodiment, a sensor fusion model (e.g., model including anextended Kalman filter, neural network model, regression model,classification model, etc.) can be used to detect the standby eventbased on the sensor signals. In a second embodiment of this variation,detecting unexpected spatial sensor signal change includes detecting aspatial sensor signal change (e.g., an IMU change) above a predeterminedthreshold value. The spatial sensor signal can be indicative ofacceleration along an axis, velocity along an axis, angular change(e.g., yaw, pitch, roll, etc.), or be indicative of any other suitableaerial system motion and/or position. In a first example of thisembodiment (shown in FIG. 18), the unexpected orientation sensor signalchange is detected when the aerial system pitch exceeds a threshold rateof change or threshold angular change (e.g., as determined fromaccelerometer and/or gyroscope signals). In a second example of thisembodiment, a first unexpected spatial sensor signal change below thepredetermined threshold value is not recognized as a standby event, butrather as a wind perturbation event, and the lift mechanism iscontrolled to correct for the wind perturbation. In this second example,a second unexpected spatial sensor signal change is above thepredetermined threshold value, and is recognized as a standby event. Ina third example of this embodiment, the standby event is detected 100S18based on a combination of an unexpected spatial sensor signal changeabove the predetermined threshold value and a supplementary signal(e.g., temperature exceeding the ambient environment temperature by athreshold amount, compressive force on the aerial system body exceedinga threshold force, etc.). In a fourth example of this embodiment, thestandby event is detected 100S18 when a pattern of spatial sensor signalchange substantially matches a predetermined pattern associated with thestandby event and/or does not substantially match predetermined patternsassociated with other flight events (e.g., wind perturbation). However,the standby event can be otherwise detected.

In a second variation, detecting the standby event 100S18 includesdetermining that the aerial system 12 has remained within a thresholdangular range from a gravity vector and/or expected orientation vector(e.g., tilted from horizontal and/or from an expected aerial systemorientation by more than 35°, 45°, 60°, etc.) for a predetermined periodof time (e.g., greater than 100 ms, 350 ms, 1 s, 2 s, etc.). Forexample, the standby event can be detected when the aerial system 12(e.g., major plane of the aerial system) has been tilted more than 45°from horizontal and/or from the target aerial system orientation formore than 1 second. However, the standby event can be otherwisedetected.

In a third variation, detecting the standby event 100S18 includesdetecting user and/or retention mechanism proximity to the aerial system12 (e.g., indicative of a user grabbing the aerial system mid-air,etc.). User and/or retention mechanism proximity can be detected usingthe aerial system's proximity sensor, touch sensor, temperature sensor(e.g., an increase in temperature), communications module (e.g., when ashort-range connection is established between the aerial system and userdevice), switches, or otherwise detected. For example, 100S18 caninclude detecting an actuation of a switch mechanically coupled to thehousing. The switch can be a button (e.g., button located convenientlyfor the retention mechanism, such as on the top or bottom housingsurface, proximal a housing perimeter, or under a fingertip when theaerial system 12 is held by a human hand), electrical contacts on thebody exterior that can be electrically connected by a conductive elementof the retention mechanism, and/or any other suitable switch.

In a fourth variation, detecting the standby event 100S18 includesreceiving an instruction (e.g., standby instruction) from a remotecontrol (e.g., user device). However, the standby event can be detected100S18 in any other suitable manner.

Operating the aerial system 12 in a standby mode 100S20 functions tosuspend aerial system flight control. The aerial system 12 is preferablyoperated in the standby mode 100S20 automatically in response todetecting the standby event 100S18 (e.g., immediately after detectingthe standby event 100S18, a predetermined time period after detectingthe standby event 100S18, after detecting the standby event 100S18 andfulfilling additional criteria, etc.). 100S20 is preferably performed bythe aerial system processor, but can additionally or alternatively beperformed by other aerial system components, a remote computing system,a user device, and/or any other suitable device.

210S22 preferably includes reducing the aerodynamic force generated bythe lift mechanism 40 to a force less than the force required for aerialsystem flight (e.g., reduced to zero force; to a fraction of therequired force, such as 1%, 5%, 10%, 50%, 75%, 1-10%, 5-25%, etc.; tojust below the required force; etc.). In a variation in which the liftmechanism 40 includes a set of rotors, the rotors can be stopped orunpowered (e.g., controlled to rotate at zero or minimal angularvelocity, provided zero or minimal power by the motors that drive them,etc.), can rotate at a slower angular velocity than when in the flightmode (e.g., a fraction of the flight mode angular velocity or minimumangular velocity required for flight, such as 1%, 5%, 10%, 50%, 75%,1-10%, 5-25%, etc.), can be otherwise altered to cooperatively generateless aerodynamic force than when in the flight mode (e.g., rotor bladeangle decreased), and/or can be controlled in any other suitable manner.For example, operating each of rotor of the set of rotors tocooperatively generate an aerodynamic force less than the flight modeaerodynamic force can include reducing power provided to each rotor toless than a power threshold required for aerial system flight (e.g., afraction of the required power, such as 1%, 5%, 10%, 50%, 75%, 1-10%,5-25%, etc.). Additionally, or alternatively, the aerodynamic forcegenerated by the lift mechanism 40 can be reduced, but not below theforce required for aerial system flight, can remain unreduced, or can bechanged in any other suitable way.

In another aspect of the present invention, with reference to FIGS.21-42, images may be presented on the remote device 14 that assist theuser 18 in controlling the aerial system 12. In the below discussion,the aerial system 12 may be controlled via user input directly onto orinto the remote device 14. However, the system 10 may also allow foruser control using user expression, e.g., thought, facial expressions,gestures, and/or voice commands.

As shown in FIG. 21, a method for aerial system control includes:selecting a first region of an imaging element 210S12, displaying animage from the first region 210S14, receiving a user input 210S16,changing a position of the aerial system 210S18, selecting a secondregion of a second imaging element 210S20, and displaying an image fromthe second region 210S22.

In operation, as shown in FIG. 22, the aerial system functions tocapture and stream video to the remote device in near-real time duringaerial system flight. The control client 16 on the remote devicefunctions to receive the video, display the video on a device output,receive commands indicative of aerial system operation instructions froma device input, and send the operation instructions to the aerial system12 in near-real time. In one variation, the operation instructions canbe received at a device input overlaid over the video, such that a usercontrols the aerial system 12 by moving the video's field of view (FOV).The aerial system 12 can receive the operation instructions andautomatically operate based on the operation instructions. In thevariation above, the aerial system 12 can be automatically operated suchthat the resultant optical sensor's FOV moves in the manner prescribedby the user 18 at the remote device 14.

In a specific example, users can directly interact with the video orimage on the client (e.g., swipe, zoom, etc.), wherein the userinteractions are automatically translated into aerial system movementsthat physically achieve the desired effect (e.g., translatinghorizontally to move the frame along the swipe vector, moving toward oraway from an object to zoom in or out, respectively, etc.). In thisspecific example, the system and method can create a WYSIWYG (what yousee is what you get)-type interaction.

The system and method for remote aerial system control can conferseveral benefits over conventional systems.

First, in some variants, the aerial system operation instructions arereceived on the video displayed by the control client 16. This allows auser 18 to control the drone through the visual output (e.g., bycontrolling what the camera is aimed at), similar to a “what you see iswhat you get” experience. The inventors have discovered that thisinteraction can be more intuitive than conventional control paradigms(e.g., having one or more joysticks that control different aspects ofaerial system operation), due to the direct link between the user input(on the video) and the video's response (e.g., created through digitalediting and/or aerial system movement). By allowing a user to controlthe aerial system (e.g., drone) in this indirect manner, the system cansimplify remote aerial system control over conventional systems.

Second, in some variants, preferably variants in which the aerial systemoperation instructions are received on the video displayed by thecontrol client 16, but additionally or alternatively variants in whichthe operation instructions are otherwise received, the control client 16can display only a cropped portion of the video on the device output,and the cropping region can be altered based on the operationinstructions and/or the aerial system status. This can allow forstabilization and/or responsiveness of the visual output (e.g., usingautomatic video stabilization programs or methods, etc.). For example,the cropping region can be moved to simulate aerial system motion whilewaiting for the aerial system to accelerate, and can be rotated tocorrect for aerial system roll. The inventors have discovered that thesemodifications of the visual output can provide a more intuitive and lessfrustrating user experience, due to the ease of watching the visualoutput and the apparent fast response of the aerial system to operationinstructions provided by the user 18.

Third, in some variants, the system can minimize aerial system massand/or volume by substituting optical sensor actuation with aerialsystem movement. For example, instead of using a three-axis gimbalsystem as the optical sensor actuation mechanism, the system can includea single-axis gimbal system, wherein optical sensor actuation about theremaining two axes can be accomplished by moving the aerial system. In aspecific example, the optical system can rotate about a singlerotational axis (e.g., the x-axis), such that the optical system is onlycapable of pitching relative to the aerial system; the aerial systemreplaces optical system yaw or roll (e.g., about the z-axis or y-axis,respectively) by rotating about an aerial system axis. The aerial system12 can additionally or alternatively function in place of or supplementthe optical system's zoom lens (e.g., by flying closer to or retreatingaway from the video subject, possibly in coordination with actuation ofthe optical zoom). By substituting the optical sensor componentfunctionalities with aerial system actions (e.g., zoom, rotation, etc.),the system enables the substituted components to be removed from thesystem, which can reduce overall system mass and/or volume.

However, the system and method can be otherwise configured, and conferany other suitable set of benefits. The control client 16 can define adisplay frame (e.g., digital structure specifying the region of theremote device output to display the video streamed from the aerialsystem 12), an input frame (e.g., digital structure specifying theregion of the remote device input at which inputs are received), or anyother suitable user interface structure. The display frame and inputframe preferably overlap, more preferably overlapping entirely (e.g.,substantially identical regions), but can alternatively be separate anddistinct, adjacent, contiguous, have different sizes, or be otherwiserelated. The control client 16 can additionally include an operationinstruction module that functions to convert inputs, received at theinput frame, into aerial system operation instructions. The operationinstruction module can be a static module that maps a predetermined setof inputs to a predetermined set of operation instructions; a dynamicmodule that dynamically identifies and maps inputs to operationinstructions; or be any other suitable module. The operation instructionmodule can calculate the operation instructions based on the inputs,select the operation instructions based on the inputs, or otherwisedetermine the operation instructions. However, the control client 16 caninclude any other suitable set of components and/or sub-modules.

The remote device 14 executing the control client 16 functions todisplay the data (e.g., as instructed by the control client 16), receiveuser inputs, compute the operation instructions based on the user inputs(e.g., as instructed by the control client 16), send operationinstructions to the aerial system 12, store control client information(e.g., associated aerial system identifiers, security keys, user accountinformation, user account preferences, etc.), or perform any othersuitable functionality. The remote device 14 can be a user device (e.g.,smartphone, tablet, laptop, etc.), a networked server system, or be anyother suitable remote computing system. The remote device 14 can includeone or more: outputs, inputs, communication systems, sensors, powersources, processing systems (e.g., CPU, memory, etc.), or any othersuitable component. Outputs can include: displays (e.g., LED display,OLED display, LCD, etc.), audio speakers, lights (e.g., LEDs), tactileoutputs (e.g., a tixel system, vibratory motors, etc.), or any othersuitable output. Inputs can include: touchscreens (e.g., capacitive,resistive, etc.), a mouse, a keyboard, a motion sensor, a microphone, abiometric input, a camera, or any other suitable input. Communicationsystems can include wireless connections, such as radios supporting:long-range systems (e.g., WiFi, cellular, WLAN, WiMAX, microwave, IR,radio frequency, etc.), short-range systems (e.g., BLE, BLE long range,NFC, Zigbee, RF, audio, optical, etc.), or any other suitablecommunication system. Sensors can include: orientation sensors (e.g.,accelerometer, gyroscope, etc.), ambient light sensors, temperaturesensors, pressure sensors, optical sensors, acoustic sensors, or anyother suitable sensor. In one variation, the remote device 14 caninclude a display (e.g., a touch-sensitive display including atouchscreen overlaying the display), a set of radios (e.g., WiFi,cellular, BLE, etc.), and a set of orientation sensors. However, theremote device 14 can include any suitable set of components.

As shown in FIG. 21, the method M210 for aerial system control includes:selecting a first region of an imaging element 210S12, displaying animage from the first region 210S14, receiving a user input 210S16,changing a position of the aerial system 210S18, selecting a secondregion of a second imaging element 210S20, and displaying an image fromthe second region 210S22. The method functions to allow a user toremotely control the aerial system 12 in an intuitive manner through theuser device. Processes of the method are preferably performedsequentially, but can alternatively be performed in parallel or in anyother suitable order. Multiple instances of the method (or portionsthereof) can be concurrently or serially performed for the same userdevice-aerial system pair. Each client (e.g., user device) is preferablyconnected to a single aerial system 12 at any given time, but canalternatively be connected to and/or control multiple aerial systemsconcurrently.

Selecting a first region of an imaging element 210S12 can function todetermine an initial or default imaging element region associated withthe aerial system 12. The imaging element can be an optical sensor(e.g., all or subset of a camera active region), image frame, image, setof images, or video sampled by the optical sensor, or be any othersuitable image- or imaging-related element. The imaging element ispreferably associated with the aerial system 12 (e.g., optical sensor ofthe aerial system, image frame of the aerial system camera, imagecaptured by the aerial system camera, etc.), but can additionally oralternatively be associated with any other suitable system. Accordingly,the first region of the imaging element can be a region of an opticalsensor (e.g., all or a subset of a camera active region), a region of animage frame, image, set of images, or video (e.g., a set of pixellocations of each image or video frame), or a region of any othersuitable image- or imaging-related element. The first region can be adefault region, or can be selected based on aerial system parameters(e.g., flight parameters such as orientation, velocity, and/or position;optical sensor status; etc.), selected by the user (e.g., preselected,selected in real time at a user device, etc.), selected based on imageprocessing (e.g., object recognition, image segmentation, etc.), and/orselected in any other suitable way. The first region is preferablyautomatically selected by the aerial system 12 (e.g., prior to imagetransmission to the remote device), but can alternatively be selected bythe remote device (e.g., based on auxiliary sensor measurements receivedfrom the aerial system 12), server system, or by any other suitablecomputing system. The region is preferably selected in near-real time(e.g., before, immediately after, or within a short duration after theimage frames are received), but can alternatively be selected at anyother suitable time.

The region preferably includes (e.g., encompasses, defines) a regulararray of pixels in a contiguous area (e.g., every pixel in the area,every second pixel in the area, every third pixel of every second row inthe area, etc.), but can alternatively include multiple non-contiguousareas, irregular pixel arrangements, and/or be any other suitableregion. For example, the region can define a rectangle (e.g., berectangular, be substantially rectangular, be rectangular except forsmall excisions removed from the rectangle and/or extensions added on tothe rectangle, etc.).

The region can be defined by a portion of one or more of the imagingelement edges (e.g., occupy the entire area of the sensor, image frame,image, etc.; border one or more of the imaging element edges; extendbetween the two horizontal edges and lie between the two vertical edges;etc.), can touch one or more of the edges (e.g., be inscribed within theimaging element edges, touch one imaging element edge with one regioncorner, etc.), or can be strictly inside of the imaging element edges(e.g., occupy a central area; occupy an area near, but not in contactwith, one or more edges; etc.). The region can be arranged symmetrically(e.g., with respect to one or more lines of symmetry of the imagingelement) or asymmetrically within the imaging element. The region canoccupy the entire area of the imaging element, or can occupy anysuitable fraction thereof (e.g., 90%, 75%, 10%, at least 50%, 40-90%,etc.). The region can include an orientation relative to the imagingelement (e.g., angle between a rectangular region edge and an imagingelement edge, orientation associated with a circular region, etc.).However, the region can alternatively have any other suitablearrangement, shape, and/or size with relation to the imaging element.Other regions selected during performance of the method (e.g., thesecond region, selected in 210S20) can have similar properties as thefirst region, different properties than the first region, and/or anyother suitable properties.

In one embodiment, regions can be automatically selected to stabilizethe video (specific example shown in FIG. 26). In a first example, theregion orientation can be selected based on the aerial systemorientation (e.g., aerial system body orientation, camera orientation,etc.; orientation relative to a gravity vector, orientation change orrate of change, etc.), which can enable camera tilt compensation. Theorientation can be determined based on aerial system sensor readings(e.g., inertial measurement unit readings such as accelerometer and/orgyroscope readings, preferably recorded concurrent with the videocapture), based on image analysis (e.g., analysis of frames of thevideo, of other images captured by an aerial system camera, of imagescaptured of the aerial system, etc.), based on user input (e.g., imagerotation request), and/or determined in any other suitable way. Forexample, the region orientation can be determined based on the aerialsystem roll angle, yaw angle, pitch angle, translation speed, relativerotor speeds, or based on any other suitable aerial system flightparameter. In a specific example of tilt compensation, a roll anglebetween an edge of the broad face of the sensor (e.g., active surfaceedge) and a projection of a gravity vector onto the broad face isdetermined. In this first specific example, the region orientationand/or change in region orientation relative to another selected regionis substantially equal in magnitude to (e.g., within 1° of, within 5°of, within 10° of, etc.) and opposite in direction of the roll angle.However, the region orientation can be otherwise determined based on theroll angle. The region can be selected to maintain a constant size(e.g., scale, resolution, etc.) of an image corresponding to the region,regardless of whether or not the aerial system is rotated, such that thesize of a compensated image is indistinguishable from the size of anuncompensated image, but can alternatively can result in different imagesizes or be otherwise selected.

The method can include receiving a video, such as a video including animage from the first region, from a camera of the aerial system 12. Thevideo can be received by a processor of the aerial system 12, by aremote computing system (e.g., user device, remote server, etc.), and/orby any other suitable system. The video frames can include only pixelsfrom the first region (e.g., be cropped to the first region), includepixels from both within and outside of the first region, or includepixels from any suitable locations of the image frame. Although a videotypically includes many frames per second, a person of skill in the artwill understand that the video can be any series of images captured bythe camera.

Displaying an image from the first region 210S14 functions to displayinformation relevant to aerial system flight to the user. The methodpreferably includes sending images (e.g., video frames), recorded by theaerial system 12 (e.g., a camera of the aerial system), to the userdevice. This can function to stream a video stream from the aerialsystem 12 to the user device. The aerial system 12 preferably sends theimages directly to the user device through a wireless connection (e.g.,a BLE connection, WiFi connection, analog RF connection, etc.), but canalternatively send the images indirectly to the user device (e.g., via aremote computing system) or otherwise send the images to the userdevice. The images (e.g., image frames) can be timestamped, associatedwith sensor measurements, such as orientation sensor measurements (e.g.,concurrently recorded sensor measurements, sensor measurements recordedwithin a threshold time period, etc.), or associated with any othersuitable information.

210S14 preferably includes displaying the image(s) in a display area,such as the display area of the user device (example shown in FIG. 6).The display area can be an entire display (e.g., display screen of theuser device), a portion of a display, can span all or portions ofmultiple displays, or can be any other suitable display area. Thedisplay is preferably a touch-sensitive display, more preferably havingan input area entirely overlapping the display area. In an example, themethod includes receiving an image sampled by a first sensor region,wherein the image includes a first image region sampled proximal a firstsensor edge and a second image region sampled proximal a second sensoredge opposing the first sensor edge. In this example, 210S14 includescontrolling the touch-sensitive display (e.g., of the user device) todisplay the first image within the entirety of a display area of thetouch-sensitive display. Further, in this example, the display areaincludes a first display edge and a second display edge opposing thefirst display edge, the first image region is displayed proximal thefirst display edge, and the second image region is displayed proximalthe second display edge (e.g., as shown in FIG. 25).

The image or images are preferably displayed (e.g., the video played) inreal- or near-real time (e.g., within a time interval of image capture,such as 1 ms, 10 ms, 20 ms, 50 ms, 1 s, etc.; wherein sending,processing, and displaying the images are performed with minimal delayafter image capture). Additionally or alternatively, the image(s) can bedisplayed after a delay interval (e.g., predetermined delay, delay basedon aerial system operation status, etc.) or after any suitable period oftime. Video frames are preferably displayed in order and for an amountof time corresponding to the capture frame rate, but alternatively maybe displayed in any suitable order and for any suitable period of time.For example, a single image can be displayed for an extended period oftime (e.g., throughout performance of the method, 1 min, 10-60 s, 1-30min, etc.). The images are preferably sent and displayed throughoutperformance of the method (e.g., continuously throughout, at intervalsthroughout, etc.), but can additionally or alternatively be sent and/ordisplayed during a specific interval, at a single time, and/or at anyother suitable time.

Receiving a user input 210S16 functions to allow a user 18 to controlthe aerial system 12. The user input preferably includes a touch input(e.g., drag input, tap input, hold input, etc.), more preferablyreceived by a touch-sensitive display. The user input is preferablyreceived on a video displayed by a user device, which functions to allowa user to interact with the video, wherein the aerial system 12 isautomatically controlled to achieve the desired video movement indicatedby the user input. The user input is preferably received by the controlclient 16, more preferably at an input area of the control client 16(e.g., the input frame, via the input device of the user device), butcan alternatively be received by any other suitable input device. Theinput area is preferably overlaid over, and encompasses substantiallythe entirety of, the display area of the client and/or user devicedisplaying the video (e.g., wherein the input area is the same size asor larger than the display area; wherein the input area overlaps 90% ormore of the display area, etc.), such that the user input can bereceived at any part of the display area. Alternatively, the input areacan be smaller than an overlay a subset of the display area, be largerthan an overlay some or all of the display area, be separate from thedisplay area, be adjacent the display area, define virtual joysticksthat control different aspects of aerial system control (e.g., separatefrom or overlapping the display area), or otherwise defined.

The user input is preferably received during image or video display.Accordingly, the user input (e.g., drag input) can be received within adisplay area concurrent with the display of the image from the firstregion in the display area. In one example, in which 210S14 includescontrolling the touch-sensitive display of the user device to displaythe first region of video, 210S16 includes receiving a drag inputincluding a translation vector (e.g., from start point to end point,from start point to intermediate hold point, from first to second holdpoint, from first to second extremum, etc.) from the touch-sensitivedisplay. In this example, the first region of a video (e.g., selected in210S12) is displayed throughout a time interval, and the drag input ispreferably received during the time interval. The translation vector candefine a horizontal component parallel to a horizontal edge of thetouch-sensitive display, a vertical component parallel to a verticaledge of the touch-sensitive display (and/or perpendicular to thehorizontal edge), and/or any other component aligned along any suitableaxis (e.g.; axis along an input plane, such as a diagonal axis; axis atan angle to the input plane; etc.).

Additionally or alternatively, the user input can include user devicemanipulation (e.g., tilting, translating, rotating, etc.), mechanicalcontrol input (e.g., joystick, button, switch, slider, etc.), audioinput (e.g., voice command, clapping input, audio location input, etc.),optical input (e.g., light signal, input discerned by image recognition,eye-tracking input, etc.), mental input (e.g., based on EEG signals),and/or any other suitable input, and can be received in any suitable wayat any suitable time. The aforementioned inputs can be determined by theuser device sensors, external sensors monitoring the user device, ordetermined by any other suitable system.

Changing a position of the aerial system 210S18 functions to effectaerial system control. Changing the position 210S18 is preferably basedon the user input, and can include generating operation instructions forthe aerial system 12 to adjust the camera angle based on the user input,which functions to interpret a desired video movement into controlinstructions for the aerial system 12 to achieve the desired videomovement (e.g., change in viewpoint, perspective, etc.).

The operation instructions are preferably generated by the controlclient 16 (e.g., by the user device running the control client 16), butcan alternatively be generated by the aerial system 12 (e.g., whereinthe user inputs and/or control instruction intermediaries are sent tothe aerial system 12), a remote computing system (e.g., wherein the userinputs and/or control instruction intermediaries are sent to the aerialsystem 12), or any other suitable computing system. The operationinstructions can be target aerial system operation parameters (e.g., 5degrees to the right, 2 meters up, etc.), aerial system controlinstructions (e.g., voltages for an aerial system component, etc.), orbe any other suitable set of instructions. In a first variation, theuser device: converts the user inputs into control instructions andsends the control instructions to the aerial system 12, wherein theaerial system operates based on the control instructions. In a secondvariation, the user device converts the user input into target operationparameters and sends the target operation parameters to the aerialsystem 12, wherein the aerial system converts the target operationparameters into control instructions and automatically operates based onthe control instructions. In a third variation, the user device sendsthe user inputs to the aerial system 12, wherein the aerial systemconverts the user inputs into control instructions and automaticallyoperates based on the control instructions. However, the operationinstructions can be otherwise generated.

Generating the operation instructions can include interpreting the userinput based on a set of predetermined relationships between the userinput parameters and aerial system actions. User input parameters caninclude: the number of concurrent, discrete inputs (e.g., touches); theduration of each continuous input; the distance of each continuousinput; the location of each input (e.g., the touch coordinates for eachtouch, beginning coordinates, end coordinates, etc.); or any othersuitable parameter. Aerial system actions can include: actuating theoptical system, yawing the aerial system 12 (e.g., about a transverseaerial system axis intersecting the optical system, about a centraltransverse axis, about an axis parallel the transverse aerial systemaxis and intersecting a target object, etc.), rolling the aerial system12 (e.g., about the longitudinal axis, about an axis normal to thecamera sensor), pitching the aerial system 12 (e.g., about the lateralaxis), translating the aerial system vertically (e.g., adjusting theaerial system height relative to a ground plane), translating the aerialsystem horizontally (e.g., adjusting the lateral relationship betweenthe aerial system 12 and a target object, moving the aerial systemcloser to or farther from the target object, etc.), or otherwiseactuating the aerial system 12. User inputs can include touch inputs,remote device manipulation (e.g., tilting, translating, etc.), or anyother suitable input. In a specific example, a single concurrent touch(e.g., single-finger drag, etc.) is mapped to aerial system and/oroptical system rotation, while multiple concurrent touches (e.g.,multi-finger drag, pinch, etc.) are mapped to aerial system translation.

The operation instructions generated based on a user input arepreferably independent from the region of the display area in which theuser input is received (e.g., such that similar user inputs received atdifferent locations within the display area have similar effects). Inone example, in which a first drag input is received at a first regionof a display area of a touch-sensitive display and defines a firsthorizontal component parallel to a horizontal edge of thetouch-sensitive display, 210S18 includes rotating the aerial system 12in a first direction about a yaw axis, based on the first horizontalcomponent. In this example, the method can further include receiving asecond drag input from the touch-sensitive display (e.g., wherein thetouch-sensitive display received the second drag input at a secondregion of the display area non-overlapping with the first region of thedisplay area), wherein the second drag input includes a secondtranslation vector codirectional with the translation vector of thefirst drag input, the second translation vector defining a secondhorizontal component codirectional with the horizontal component. Inthis example, the method can further include rotating the aerial system12 in the first direction about the yaw axis again, based on the secondhorizontal component. However, the region of the display area in which auser input is received can alternatively affect the operationinstructions generated based on that user input (e.g., a drag input in a‘yaw’ region can control aerial system yaw, while a similar drag inputin a ‘translation’ region can control aerial system translation).

The duration, distance, speed, and/or acceleration of aerial systemaction (e.g., rotation speed, translation speed, etc.) can be relatedto: the temporal duration of continuous user input, the distance orlength of continuous user input, the speed and/or acceleration of theuser input position change or any other suitable user input parameter.The duration, distance, speed, and/or acceleration of aerial systemaction can be: proportional to the user input parameter, inverselyproportional to the user input parameter, a monotonic function of theuser input parameter (e.g., increase monotonically or strictlymonotonically with the user input parameter, decrease monotonically orstrictly monotonically with the user input parameter), or be otherwiserelated. For example, 210S18 can include rotating the aerial system 12based on a value of a rotation parameter, wherein the value is amonotonically increasing function of a magnitude of an aspect of a userinput (e.g., component of a drag vector, such as vertical or horizontalcomponent). In a specific example, the rotation parameter is the aerialsystem rotation speed. The value of aerial system parameter changemapped to the user input parameter can be: constant, proportionallyscaled with the aerial system parameter value or different aerial systemparameter value (e.g., distance from the remote device), inverselyscaled with aerial system parameter value, or otherwise related. In oneexample, the same linear user input can be mapped to a first translationdistance when the aerial system is beyond a threshold distance away fromthe remote device, and mapped to a second translation distance when theaerial system 12 is within a threshold distance away from the remotedevice, wherein the first translation distance is larger than the secondtranslation distance.

In a first variation, the user inputs are indicative of field of viewrotations (e.g., turn left, turn right, turn up, turn down), wherein therotation inputs are mapped to a blend of optical system and aerialsystem rotation actions. The aerial system 12 is preferably translatedin the opposing direction as the rotation input arcuate direction, butcan alternatively be translated in the same direction.

A first embodiment of the first variation includes rotating the aerialsystem about a yaw axis based on the horizontal component (specificexamples shown in FIGS. 7-8). In this embodiment, a user input to rotatethe field of view to the left (e.g., move the camera FOV to the left;move the video perspective to the left; change the horizontalperspective to the left but not the vertical perspective; etc.) ismapped to control instructions to yaw the aerial system counterclockwise(e.g., positive yaw; counterclockwise as viewed from above relative to agravity vector, aerial system top, camera top, etc.) about a yaw axis,and/or a user input to rotate the field of view to the right (e.g., movethe camera FOV to the right; move the video perspective to the right;change the horizontal perspective to the right but not the verticalperspective; etc.) is mapped to control instructions to yaw the aerialsystem clockwise (e.g., negative yaw; clockwise as viewed from aboverelative to a gravity vector, aerial system top, camera top, etc.) abouta yaw axis (e.g., same or different yaw axis). The yaw axis canintersect the aerial system or be external the aerial system 12. Whenthe yaw axis is external the aerial system 12, the system generating thecontrol instructions can automatically: determine the target objectdistance from the aerial system 12, determine the yaw axis locationrelative to the aerial system 12 based on the target object distance,and determine the control instructions based on the yaw axis location(e.g., transform or shift the control instructions accordingly).Alternatively, the yaw axis distance from the aerial system 12 can besubstantially constant, wherein the aerial system yaws along the samearc angle, irrespective of the target object distance from the aerialsystem 12. However, the left and/or right rotation user input can beotherwise mapped.

The left rotation user input can include: a single-touch hold (e.g.,temporally continuous signal) and drag (e.g., a series of temporallycontinuous signals, each (or substantially all of, a majority of, etc.)temporally successive signal further along a positive x-axis than thelast signal, etc.) to the right of the display area and/or input area(e.g., toward a right vertical edge of the display); a touch, hold, anddrag to the left of the display area and/or input area; a single-touchhold on the left of the display area and/or input area; a series ofconsecutive taps on the left of the display area and/or input area; aseries of consecutive taps on the right of the display area; user deviceroll counterclockwise (e.g., as determined by the user deviceorientation sensor); or include any other suitable input pattern withany other suitable set of parameters. For example, the left rotationuser input can include a translation vector that defines a horizontalcomponent that points right, and in response, the aerial system 12 canrotate about a yaw axis counterclockwise as viewed from above (e.g.,relative to a gravity vector, etc.).

The right rotation user input can include: a single-touch hold and drag(e.g., a series of temporally continuous signals, each (or substantiallyall of, a majority of, etc.) temporally successive signal further alonga negative x-axis than the last signal, etc.) to the left of the displayarea and/or input area (e.g., toward a left vertical edge of thedisplay); a touch, hold, and drag to the right of the display areaand/or input area; a single-touch hold on the right of the display areaand/or input area; a series of consecutive taps on the right of thedisplay area and/or input area; a series of consecutive taps on the leftof the display area; user device roll clockwise (e.g., as determined bythe user device orientation sensor); or include any other suitable inputpattern with any other suitable set of parameters. For example, theright rotation user input can include a translation vector that definesa horizontal component that points left, and in response, the aerialsystem 12 can rotate about a yaw axis clockwise as viewed from above(e.g., relative to a gravity vector, etc.).

In a second embodiment of the first variation (specific examples shownin FIGS. 9-10), the aerial system 12 includes a gimbal system (e.g.,single-axis gimbal system) rotatably mounting the optical sensor 36 tothe body 20. For example, the optical sensor 36 (e.g., camera) can berotatable about a gimbal axis, wherein the gimbal axis is substantially(e.g., within 1°, within 5°, within 10°, etc.) perpendicular the yawaxis and/or roll axis, and/or substantially (e.g., within 1°, within 5°,within 10°, etc.) parallel the optical sensor active surface (e.g.,camera sensor). The optical system angular position and/or gimbalposition can be determined based on: user inputs, desired scene changes,the aerial system pitch angle (e.g., wherein the optical system angularposition can be dynamically changed to counteract changes in the sampledscene due to aerial system pitch), or otherwise determined. In oneexample, a user input to rotate the field of view upward (e.g., move thecamera FOV upward; move the video perspective to upward; change thevertical perspective upwards but not the horizontal perspective; etc.)is mapped to control instructions to control instructions to pitch theoptical sensor upward about the gimbal axis, and/or a user input torotate the field of view downward (e.g., move the camera FOV downward;move the video perspective downward; change the vertical perspectivedownwards but not the horizontal perspective; etc.) is mapped to controlinstructions to pitch the optical sensor downward about the gimbal axis.

The upward rotation user input can include: a single-touch hold and drag(e.g., a series of temporally continuous signals, each (or substantiallyall of, a majority of, etc.) temporally successive signal further alonga negative y-axis than the last signal, etc.) to the bottom of thedisplay area and/or input area (e.g., toward a lower horizontal edge ofthe display); a touch, hold, and drag to the top of the display areaand/or input area; a single-touch hold on the top of the display areaand/or input area; a series of consecutive taps on the top of thedisplay area and/or input area; a series of consecutive taps on thebottom of the display area; user device pitch forward (e.g., asdetermined by the user device orientation sensor); or include any othersuitable input pattern with any other suitable set of parameters. Forexample, the upward rotation user input can include a translation vectorthat defines a vertical component (e.g., perpendicular to the horizontaledge of the display) that points down, and in response, the camera canrotate upward about the gimbal axis based on the vertical component.

The downward rotation user input can include: a single-touch hold anddrag (e.g., a series of temporally continuous signals, each (orsubstantially all of, a majority of, etc.) temporally successive signalfurther along a positive y-axis than the last signal, etc.) to the topof the display area and/or input area (e.g., toward an upper horizontaledge of the display); a touch, hold, and drag to the bottom of thedisplay area and/or input area; a single-touch hold on the bottom of thedisplay area and/or input area; a series of consecutive taps on thebottom of the display area and/or input area; a series of consecutivetaps on the top of the display area; user device pitch backward, towardthe user (e.g., as determined by the user device orientation sensor); orinclude any other suitable input pattern with any other suitable set ofparameters. For example, the downward rotation user input can include atranslation vector that defines a vertical component (e.g.,perpendicular to the horizontal edge of the display) that points up, andin response, the camera can rotate downward about the gimbal axis basedon the vertical component.

In a third embodiment of the first variation, a user input to rotate thefield of view upward (e.g., move the camera FOV upward; move the videoperspective to upward; change the vertical perspective upwards but notthe horizontal perspective; etc.) is mapped to control instructions topitch the optical system upward about a rotational axis, and/or a userinput to rotate the field of view downward (e.g., move the camera FOVdownward; move the video perspective downward; change the verticalperspective downwards but not the horizontal perspective; etc.) ismapped to control instructions to pitch the optical system downwardabout a rotational axis. Alternatively, the upward rotation user inputcan be mapped to pitch the aerial system upward about a pitch axis(e.g., lateral axis, axis parallel the lateral axis, etc.), or beotherwise mapped, and/or the downward rotation user input can be mappedto pitch the aerial system downward about a pitch axis (e.g., lateralaxis, axis parallel the lateral axis, etc.), or be otherwise mapped. Theupward and/or downward rotation user input can be the same as in thesecond embodiment, and/or can be any other suitable input.

In a fourth embodiment of the first variation, a user input to rotatethe field of view about an axis normal to the field of view can bemapped to control instructions to roll the aerial system about a rollaxis (e.g., longitudinal axis, axis parallel the longitudinal axis,etc.), control instructions to rotate the cropped region of the image,or be otherwise mapped. The roll rotation user input can include: atwo-touch hold and drag along a substantially arcuate path; a singletouch hold and drag along a substantially arcuate path; user device yaw(e.g., as determined by the user device orientation sensor); or anyother suitable input pattern with any other suitable set of parameters.The aerial system preferably rolls in the angular direction of thearcuate path, but can alternatively roll in the opposing direction.

In a second variation, the user inputs are indicative of field of viewtranslations (e.g., move left, move right, move up, move down), whereinthe translation inputs are mapped to aerial system translation actions.The aerial system is preferably translated in the same direction as thetranslation input axis, but can alternatively be translated in theopposing direction. The translation user inputs can be different fromthe rotation user inputs, but can alternatively be substantiallysimilar. In one example of the latter variation, the same input can bemapped to a FOV rotation action until an input threshold is reached, atwhich point the input is mapped to a FOV translation action. In aspecific example, a continuous single-touch hold on the left of thedisplay can rotate the FOV leftward, up to a 180° rotation, at whichpoint the input is remapped to aerial system translation to the left.However, the translation inputs can be otherwise related to the rotationinputs.

In a first embodiment of the second variation, a user input to translatethe field of view horizontally (e.g., laterally translate the cameraFOV; laterally translate the video perspective; translate the horizontalperspective but not change the vertical perspective; etc.) can be mappedto: control instructions to translate the aerial system 12 (e.g., alongan x-axis) along a lateral translation axis (e.g., parallel orcoincident the central lateral axis of the body 20; perpendicular agravity vector and parallel the optical sensor active surface;substantially parallel and/or perpendicular these or other references,such as within 1°, 5°, or 10°; etc.), to translate the optical systemalong the lateral translation axis, or to any other suitable aerialsystem action. However, the lateral translation user input can beotherwise mapped. The lateral translation user input can include: atwo-touch hold and linear lateral drag on the display area and/or inputarea (e.g., toward a vertical edge of the display); a single-touch holdon one side of the display area and/or input area; a series ofconsecutive taps on one side of the display area and/or input area; userdevice roll; or include any other suitable input pattern with any othersuitable set of parameters. For example, the lateral translation userinput can include a translation vector that defines a horizontalcomponent parallel to a horizontal edge of the display, and in response,the aerial system can translate in a direction substantially parallel(e.g., within 1°, 5°, 10°, etc.) to a broad face of the optical sensor(e.g., camera sensor active surface), based on the horizontal component.In a first specific example (e.g., as shown in FIGS. 11 and 21), thehorizontal component points right and the aerial system translates left.In a second specific example (e.g., as shown in FIGS. 12 and 22), thehorizontal component points left and the aerial system translates right.However, the translation directions can be reversed, or the input can bemapped to any other suitable translation.

In a second embodiment of the second variation (specific examples shownin FIGS. 13, 14, 17, and 18), a user input to translate the field ofview vertically (e.g. vertically translate the camera FOV; verticallytranslate the video perspective; translate the vertical perspective butnot change the horizontal perspective; etc.) can be mapped to: controlinstructions to translate the aerial system (e.g., along an y-axis)along a vertical translation axis (e.g., parallel or coincident thecentral vertical axis of the body, parallel or coincident the aerialsystem yaw axis, parallel a gravity vector, substantially paralleland/or perpendicular these or other references, such as within 1°, 5°,or 10°, etc.), to translate the optical system along the verticaltranslation axis, or to any other suitable aerial system action.However, the vertical translation user input can be otherwise mapped.The vertical translation user input can include: a two-touch hold andlinear longitudinal drag on the display area and/or input area (e.g.,along a y-axis of the display area or input area); a single-touch holdon one end of the display area and/or input area (e.g., top end, bottomend); a series of consecutive taps on one end of the display area and/orinput area; user device pitch; or include any other suitable inputpattern with any other suitable set of parameters.

In a third variation, the user inputs are indicative of image scaleadjustments, wherein the image scale user inputs are mapped to aerialsystem translation actions. The aerial system preferably moves away fromthe target object (e.g., away from the object currently within the FOV;against a normal vector of the active face of the camera; etc.) inresponse to a zoom out user input, and moves closer to the target object(e.g., toward the object currently within the FOV; along a normal vectorof the active face of the camera; etc.) in response to a zoom in userinput; specific examples shown in FIGS. 15 and 16. Additionally oralternatively, the user input can be mapped to a blend of optical systemzoom and aerial system translation. In one example, in response toreceipt of a zoom in user input, the aerial system moves toward thetarget object up to a threshold distance away from the target object.Upon achieving the threshold distance and receiving further inputsindicative of zooming in, the inputs can be automatically remapped tozooming in the camera (e.g., through digital zoom). When a zoom out userinput is received, the camera is zoomed out until the maximum camerafocal length is achieved, at which point the user input is remapped toaerial system translation (e.g., translation away from the targetobject). The threshold distance can be predetermined, set by the user 18(e.g., entered into the control client 16), or otherwise determined. Theimage scale user input can include: a pinching movement (e.g., two-touchmoving toward each other), indicative of zooming in; an expandingmovement (e.g., two-touch moving away from each other), indicative ofzooming out; a sliding movement (e.g., sliding up, indicative of zoomingin; sliding down, indicative of zooming out, etc.); or any othersuitable user input.

In a fourth variation, the user inputs are indicative of image scaleadjustments, wherein the image scale user inputs are mapped to aerialsystem translation actions. The aerial system preferably moves away fromthe target object (e.g., away from the object currently within the FOV;against a normal vector of the active face of the camera; etc.) inresponse to a zoom out user input, and moves closer to the target object(e.g., toward the object currently within the FOV; along a normal vectorof the active face of the camera; etc.) in response to a zoom in userinput; specific examples shown in FIGS. 19 and 20. Additionally oralternatively, the user input can be mapped to a blend of optical systemzoom and aerial system translation. In one example, in response toreceipt of a zoom in user input, the aerial system moves toward thetarget object up to a threshold distance away from the target object.Upon achieving the threshold distance and receiving further inputsindicative of zooming in, the inputs can be automatically remapped tozooming in the camera (e.g., through digital zoom). When a zoom out userinput is received, the camera is zoomed out until the maximum camerafocal length is achieved, at which point the user input is remapped toaerial system translation (e.g., translation away from the targetobject). The threshold distance can be predetermined, set by the user(e.g., entered into the control client 16), or otherwise determined. Theimage scale user input can include: a pinching movement (e.g., two-touchmoving toward each other), indicative of zooming out; an expandingmovement (e.g., two-touch moving away from each other), indicative ofzooming in; a sliding movement (e.g., sliding up, indicative of zoomingout; sliding down, indicative of zooming in, etc.); or any othersuitable user input.

210S18 can include mapping a user input to multiple aerial systemmovements (e.g., as shown in FIGS. 23 and 24). For example, based on auser input corresponding to multiple directional components (e.g.,orthogonal components such as a horizontal component and a verticalcomponent), the aerial system 12 can perform a movement corresponding toeach directional component. In a first specific example, a one-fingerdiagonal drag input pointing up and to the right is mapped to both apositive aerial system yaw and a downward camera rotation about a gimbalaxis. In a second specific example, a two-finger diagonal drag inputpointing down and to the right is mapped to an aerial system translationup and to the left. However, the user input can be otherwise mapped tomultiple directional commands. In a specific example, a leftwardexpanding movement (e.g., wherein both finger contact points move leftwhile moving away from each other, wherein the midpoint between thefinger contact points moves left, etc.) is mapped to an aerial systemtranslation both rightward and toward the target object.

The aerial system can perform the movements concurrently, at overlappingtimes, consecutively, in alternating steps (e.g., performing a firststep of a yaw movement, then a first step of an elevation increasemovement, then a second step of the yaw movement, and then a second stepof the elevation increase movement), or with any other suitable timing.Additionally or alternatively, a user input can be mapped to a singleaerial system movement (e.g., a predominantly horizontal user inputbeing mapped only to a horizontal aerial system movement, such as yaw orhorizontal translation; a predominantly vertical user input being mappedonly to a vertical aerial system movement, such as aerial system orcamera pitch or vertical translation), or can be interpreted in anyother suitable way.

Additionally or alternatively, some or all of the user inputs can bedetermined based on remote control (e.g., user device) position,orientation, and/or movement. In a first specific example, tilting theuser device forward can be mapped to zooming in and/or translating theaerial system 12 toward the target object, and/or tilting the userdevice backward can be mapped to zooming out and/or translating theaerial system away from the target object. In a second specific example,tilting the user device to the left can be mapped to the aerial systemtranslating to the left, and/or tilting the user device to the right canbe mapped to the aerial system translating to the right. In a thirdspecific example, translating the user device to the left can be mappedto the aerial system undergoing a positive yaw, and/or translating theuser device to the right can be mapped to the aerial system undergoing anegative yaw. However, any suitable user inputs can be mapped to anysuitable aerial system movements.

Selecting a second region of a second imaging element region 210S20 canfunction to compensate for aerial system operation. The second imagingelement region is preferably the same imaging element region used in210S12 (e.g., both regions of the same video, both regions of the samecamera sensor, etc.), but alternatively can be an imaging element regiontemporally preceding the first imaging element region (e.g., beyond orwithin a threshold time duration), temporally following the firstimaging element region (e.g., beyond or within a threshold timeduration), spatially adjacent or overlapping the first imaging elementregion (e.g., recorded concurrently with the imaging element region orrecorded at any other suitable time), associated with (e.g., of orcaptured by) a second optical sensor on the same or different aerialsystem 12, or be any other suitable image- or imaging-related element.The second region is preferably selected by the same system as thesystem that selects the first region in 210S12, but additionally oralternatively can be selected by any other suitable computing system.The second region is preferably selected in near-real time (e.g.,before, immediately after, or within a short duration after the imageframes are received), similar to the first region, but can alternativelybe selected at any other suitable time. The second region is preferablyselected 210S20 concurrently with changing a position of the aerialsystem 210S18, but can additionally or alternatively be selected before210S18, after 210S18, or at any other suitable time. The second regionpreferably defines a similar area (or areas) as the first region (e.g.,a regular array of pixels in a contiguous area, such as an area defininga rectangle), but can alternatively include multiple non-contiguousareas, irregular pixel arrangements, and/or be any other suitableregion. Similar to the first region, the second region can have anysuitable arrangement, shape, and/or size with relation to the imagingelement from which it is selected. The first and second regions can bethe same or different sizes, can have the same or different orientation,and can be in the same position as each other, overlap (e.g., sharepixels), be contiguous, or be entirely separate. Although 210S20 refersto a second region, a person skilled in the art will understand that theregion can be any suitable region (e.g., third region, fourth region,etc.).

210S20 preferably includes compensating for aerial system response lag.For example, aerial system response lag (e.g., behind user input) can bedue to an operation instruction generation delay, signal transmissiondelay, propeller spool up time, aerial system acceleration time (e.g.,to begin aerial system movement, to halt or slow aerial system movement,etc.), delays due to the aerodynamics of flight, or delay due to anyother suitable effect. Compensating for aerial system response lag caninclude determining a translation vector associated with the user inputand selecting the second region such that the second region is displacedalong the translation vector from the first region. The translationvector can be a display vector (e.g., vector from start point to endpoint or end point to start point of a drag input received on thedisplay, vector aligned along a display axis, etc.), an aerial systemvector (e.g., vector along or opposing aerial system motion associatedwith the user input, vector aligned along an aerial system or opticalsensor axis, vector along an optical sensor active surface associatedwith a drag input on an image sampled by the optical sensor, etc.), orbe any other suitable vector, defined relative to any other suitablesystem component.

In one variation, the first and second regions cooperatively define atranslation vector from the center of the first region to the center ofthe second region. The variation can additionally or alternativelyinclude moving the camera in a direction substantially opposing (e.g.,within 1°, 5°, 10°, etc.) the translation vector (e.g., in 210S18). In afirst embodiment of the variation, a camera sensor includes a firstsensor edge and a second sensor edge opposing the first sensor edge, andthe first and second regions are regions of the camera sensor. In thisembodiment, the translation vector defines a horizontal componentperpendicular to the first sensor edge, and 210S18 includes translatingthe aerial system in a flight direction at an obtuse (e.g.,substantially straight, such as within 1°, 5°, 10°, etc.) or straightangle to the horizontal component. In one example of this embodiment,the flight direction is substantially perpendicular (e.g., within 1°,5°, 10°, etc.) a gravity vector and substantially parallel (e.g., within1°, 5°, 10°, etc.) a broad face of the camera sensor (e.g., the activesurface).

A specific example further includes: receiving a first image sampled bythe first region of the camera sensor (first sensor region), the firstimage including a first image region sampled proximal the first sensoredge and a second image region sampled proximal the second sensor edge;displaying the first image within the entirety of a display area of atouch-sensitive display, the display area including a first display edgeand a second display edge opposing the first display edge, the firstimage region displayed proximal the first display edge and the secondimage region displayed proximal the second display edge; concurrent withthe display of the first image (e.g., first image displayed at somepoint during the drag input, first image displayed throughout the entiredrag input, etc.), receiving a drag input from the touch-sensitivedisplay, the drag input received within the display area, the drag inputincluding a drag vector extending toward the second display edge (e.g.,and away from the first display edge); and selecting a second region ofthe camera sensor (second sensor region) based on the drag input,wherein the center of the first region is more proximal the secondsensor edge (e.g., and more distal the first sensor edge) than thecenter of the second region.

In an example of compensating for aerial system response lag, wherein avideo is received from the aerial system camera (e.g., wherein theregions are selected from the video, selected from the camera sensorthat samples the video, etc.), 210S12 can include cropping out one ormore edges of every image frame (of the video); and in response toreceipt of a user input 210S16 indicative of aerial system movement in afirst direction, 210S20 can include cropping out less of a first imageedge, and cropping out more of a second image edge. This can function togive the appearance of video frame motion until the aerial systemcatches up with (e.g., performs) the specified action. In one example,each videoframe is cropped to show only the center of the frame. Inresponse to receipt of a lateral user input to translate the aerialsystem to the left, each subsequent videoframe can be cropped to show aframe region left of center. Upon aerial system translation (e.g., asdetermined from the aerial system accelerometer or gyroscope),subsequent videoframes can be cropped to show only the frame centeragain. The selected region can be adjusted gradually (e.g., moving anincremental amount for each of several video frames, such as all thevideo frames to be displayed during the expected delay period), adjustedsuddenly, adjusted based on the user input speed (e.g., tracking a draginput, such as moving an image frame region to maintain its positionunder the finger used for the drag input), or adjusted at any othersuitable rate.

Compensating for the aerial system response lag can additionally oralternatively include recovering from a second region selection (e.g.,re-centering the region selection, shifting the displayed region back tothe first region, etc.). The recovery type, direction, speed,acceleration, or other recovery parameter can be predetermined,automatically determined based on aerial system parameters (e.g., flightspeed, angular position, target position, etc.), or otherwisedetermined. In one variation, after selecting a second region displacedfrom the first region in a first direction, the method can includeselecting a third region displaced from the second region in a seconddirection, wherein the first and second directions cooperatively form anobtuse or straight angle (e.g., antiparallel; oppose each other;substantially oppose each other, such as within 1°, 5°, 10°, etc.; havevertical and/or horizontal components that oppose each other; etc.). Forexample, the center of the third region can be between the centers ofthe first and second regions, can be substantially coincident the centerof the first region (e.g., wherein the third region substantiallyoverlaps the first region, wherein the third region is rotated relativeto the first region, etc.), can oppose the center of the second regionacross the center of the first region, or can have any other suitableposition relative to the other regions. The centers can be consideredsubstantially coincident if they are within a threshold distance (e.g.,5, 10, or 50 pixels; 1, 2, or 10 mm; etc.) or within a thresholdfraction (e.g., 1%, 5%, 10%, etc.) of a dimension associated with theregion (e.g., a length or width of the region or the imaging element theregion is selected from).

In a first embodiment of this variation, the region selection isrecovered during the aerial system movement (e.g., recovered graduallythroughout the movement). For example, the method can additionallyinclude, while rotating the aerial system about the yaw axis based on auser input: receiving a third video; selecting a third region of thethird video, wherein the third region is between the first and secondregions (e.g., not overlapping either region, overlapping one or bothregions); and displaying the third region of the third video (e.g., atthe touch-sensitive display). This example can further include (e.g.,still during the aerial system rotation, after the aerial systemrotation, after displaying the third region of the third video, etc.)selecting a first region of the fourth video, wherein the first regionof the fourth video is in the same position as the first region of thefirst video, and displaying the first region of the fourth video (e.g.,at the touch-sensitive display). In a specific example, after displayingthe second region of the second video, subsequent intermediate regionsare selected and displayed to gradually move the region selection backtoward the first region (e.g., the center region).

In a second embodiment of this variation, the region selection isrecovered during or temporally near (e.g., within 1, 2, or 10 s, etc.) achange in the aerial system movement (e.g., return to hovering, movementdirection change, etc.). This can function to compensate for lagassociated with the aerial system movement change (e.g., lag inresponding to a stop command, acceleration lag, etc.). For example,after selecting a second region rotated in a first direction relative tothe first region and beginning to translate the aerial system (e.g., ina direction substantially parallel to the camera sensor active surface),the method can additionally include: receiving a third video from thecamera; selecting a third region of the third video (e.g., wherein thecenter of the third region is substantially coincident with the centerof the first region, wherein the third region is rotated relative to thefirst region in a direction opposite the first direction); anddisplaying the third region of the third video (e.g., at thetouch-sensitive display). In a first specific example, during most ofthe aerial system movement, regions near the second region are selectedand displayed, and near the end of aerial system movement, regionscloser to the first region are selected to compensate for the aerialsystem deceleration time. In a second specific example, intermediateregions are selected and displayed to gradually move the regionselection back toward the first region during the aerial systemmovement, and near the end of aerial system movement, regions past thefirst region are selected to compensate for the aerial systemdeceleration time (after which, further intermediate regions can beselected and displayed to again gradually move the region selection backtoward the first region). However, the aerial system response lag can beotherwise compensated for.

210S20 can additionally or alternatively include compensating for aerialsystem rotation (e.g., as described in 210S12), which can occur when theaerial system rotates or translates. This functions to stabilize theresultant image, which would otherwise appear to rotate with the aerialsystem 12. For example, the aerial system 12 can roll when rotatinglaterally or translating horizontally. In a second example, the aerialsystem 12 will pitch when moving forward or backward. Compensating foraerial system rotation preferably includes selecting the orientation ofthe second region. For example, the second region can be rotatedrelative to the first region (e.g., by an amount substantially equal toan aerial system roll angle, such as within 1°, 5°, or) 10°. In aspecific example, a second camera sensor region is rotated about a rollaxis in a first direction relative to a first camera sensor region, theroll axis normal to a broad face of the camera sensor (e.g., activesurface), and an aerial system translation movement includes rotatingthe aerial system about the roll axis in a second direction opposite thefirst direction. In this specific example, the second region ispreferably substantially upright (e.g., an edge of the second regionsubstantially aligned with a gravity vector, such as within 1°, 5°, or10°) during the capture of images (e.g., to be displayed in 210S22) fromthe second region.

In a first variation of 210S20, the second region is selected based onthe user input, preferably concurrent with changing a position of theaerial system based on the user input. In a first example, the userinput is a drag input including a translation vector (e.g., vector fromstart point to end point; component of a vector associated with the draginput, such as a horizontal or vertical component; etc.), and the secondregion is translated along the translation vector relative to the firstregion. In a second example, the user input is a drag input receivedwithin a display area of a touch-sensitive display, the display areaincluding (e.g., bordered by) a first display edge and a second displayedge opposing the first display edge. In this example, the drag inputincludes a drag vector (e.g., vector from start point to end point)extending away from the first display edge toward the second displayedge. Furthermore, in this example, an image sampled by a first sensorregion is displayed within the entirety of the display area (e.g.,concurrent with receiving the drag input), wherein the image includes afirst image region sampled proximal a first sensor edge and a secondimage region sampled proximal a second sensor edge opposing the firstsensor edge, the first image region is displayed proximal the firstdisplay edge, and the second image region is displayed proximal thesecond display edge. In this example, the second region is selectedbased on the drag input such that the center of the first region is moreproximal the second sensor edge than the center of the second region.

In a first embodiment of this variation, in response to a user input tomove the field of view laterally (e.g., rotate left, translate left,rotate right, translate right), the second region is displaced laterallyrelative to the first region. In a first specific example of thisembodiment (e.g., as shown in FIG. 25), in which the regions areselected from a camera sensor, the second region is selected to the leftof the first region as viewed facing toward the camera (e.g., viewpointwithin the camera field of view) in response to a user input to move thefield of view left. In a second specific example of this embodiment, thesecond camera sensor region is selected to the right of the first camerasensor region as viewed facing toward the camera in response to a userinput to move the field of view right. In a third specific example ofthis embodiment, in which the regions are selected from a video, image,or image frame, the second region is selected to be to the left of thefirst region in response to a user input to move the field of view left.In a fourth specific example of this embodiment, in which the regionsare selected from a video, image, or image frame, the second region isselected to be to the right of the first region in response to a userinput to move the field of view right.

In a second embodiment of this variation, in response to a user input tomove the field of view vertically (e.g., rotate up, translate up, rotatedown, translate down), the second region is displaced verticallyrelative to the first region. In a first specific example of thisembodiment, in which the regions are selected from a video, image, orimage frame, the second region is selected to be above the first regionin response to a user input to move the field of view up. In a secondspecific example of this embodiment, in which the regions are selectedfrom a video, image, or image frame, the second region is selected to bebelow the first region in response to a user input to move the field ofview down. In a third specific example of this embodiment, in which theregions are selected from a camera sensor and the image formed on thecamera sensor is flipped (e.g., due to the camera optics) relative tothe objects forming the image, the second region is selected to be belowthe first region in response to a user input to move the field of viewup. In a fourth specific example of this embodiment, in which theregions are selected from a camera sensor and the image formed on thecamera sensor is flipped (e.g., due to the camera optics) relative tothe objects forming the image, the second region is selected to be abovethe first region in response to a user input to move the field of viewdown.

In a second variation of 210S20, the second region is selected based onan aerial system position change. This can function to stabilize thevideo stream. The position change can be an intentional change (e.g.,aerial system roll or pitch to enable lateral translation),unintentional change (e.g., movement due to wind or aerial systemcollision), or any other suitable position change. However, the secondregion can additionally or alternatively be selected in any othersuitable way, based on any other suitable criteria, at any othersuitable time.

The second region is preferably selected by the aerial system 12. In oneexample, receiving the first video and second videos and selecting thefirst and second regions is performed by a processor on-board the aerialsystem 12. Additionally or alternatively, the second region can beselected at a remote computing system such as the user device, or at anyother suitable system.

Displaying an image from the second region 210S22 can function to streamthe compensated images. The images can be displayed as described in210S14, or in any other suitable manner. For example, different pixellocation subsets of the image frame can be selected as the first andsecond regions (e.g., wherein the corresponding pixels of the first andsecond regions are displayed at the touch-sensitive display during210S14 and 210S22, respectively).

The image from the second region is preferably displayed concurrent with210S18. The image can be displayed at the start of aerial system motion,during aerial system motion, at the end of aerial system motion, and/orduring the entirety of aerial system motion. In one example, the secondregion of the second video is displayed concurrent with changing theaerial system position based on the drag input. Additionally oralternatively, the image can be displayed before 210S18, after 210S18,and/or at any other suitable time. In one example, the image from thesecond region and the image from the first region are sub-images of thesame image. However, the image from the second region can be otherwisedisplayed.

The method can optionally include compensating for aerial systemrotation using a gimbal system (e.g., active or passive), attached tothe optical system, that automatically stabilizes the image (e.g.,relative to a gravity vector). In one example, the gimbal system can bemulti-axis (e.g., 3-axis), wherein each gimbal (e.g., roll, pitch, andyaw) includes a resolver. The resolvers receive a gyro output (e.g.,indicative of gyro deviation from null), perform an automatic matrixtransformation according to each gimbal angle, and deliver the requiredtorque to the respective drive motor connected to each gimbal. However,the aerial system rotation can be otherwise compensated for.

Aerial system components other than the lift mechanism preferablycontinue to operate in the standby mode, but can alternatively be turnedoff (e.g., unpowered), operated in a standby state, or otherwiseoperated. For example, the sensors and processor can continue to detectand analyze aerial system operation parameters and/or determine theaerial system operation status (e.g., spatial status, flight status,power status, etc.), and the communication system can continue totransmit data (e.g., video stream, sensor data, aerial system status,etc.) from the aerial system (e.g., to a user device, remote computingsystem, etc.). Continuing to detect and analyze aerial system status canenable the system to detect a flight event 100S10 while operating in thestandby mode 100S20. For example, this enables repetition of the method(e.g., re-entering a flight mode, etc.) when the aerial system isreleased after being grabbed and retained (e.g., and entering a standbymode in response). In addition, this can enable flight mode recoverywhen an event is incorrectly identified as a standby event. In aspecific example, in which a wind perturbation or collision with anobject is misidentified as a grab event, the aerial system can enter thestandby mode and begin to freefall (as it is not actually supported by aretention mechanism). In this specific example, the freefall can then bedetected, and the aerial system 12 can resume operation in the flightmode 100S12 in response to freefall detection. Alternatively, operatingthe aerial system 12 in the standby mode 100S20 can include turning offand/or reducing the power consumption of any or all of the other aerialsystem components, operating the aerial system components at anysuitable power consumption level in any suitable manner, and/or 100S20can be performed in any other suitable manner.

Although omitted for conciseness, the preferred embodiments includeevery combination and permutation of the various system components andthe various method processes, wherein the method processes can beperformed in any suitable order, sequentially or concurrently.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

What is claimed is:
 1. An aerial system, comprising: a body; a liftmechanism coupled to the body; an orientation sensor coupled to the bodyfor sensing an orientation of the body; an optical system controllablymounted to the body by an actuation system; and a processing systemcoupled to the lift mechanism, the optical system, and the actuationsystem, the processing system configured to: detect a release event inwhich the aerial system is released from being physically held by auser; enter a hover idle state in which the aerial system searches for atarget person; recognize the target person; track the target person andscan for user expressions; identify a hand of the target person; detecta first predefined user expression including an open palm gesture of thehand of the target person and responsively track a movement of the handin the open palm gesture and execute a first corresponding eventincluding operating the lift mechanism to fly the aerial system in thedirection of the tracked movement of the hand in the open palm gesturewhile maintaining the target person within a field of view of theoptical system; detect a second predefined user expression including a“V” sign gesture by the hand of the target person and responsivelyexecute a second corresponding event including operating the opticalsystem to take a picture of the target person; enter the hover idlestate upon completing the second corresponding event; detect a standbyevent including a change in an orientation vector of the body uponsensing a change in an orientation sensor signal received from theorientation sensor; and responsively operate the lift mechanism in astandby mode.
 2. An aerial system, as set forth in claim 1, furthercomprising additional sensor(s) associated with the aerial systemconfigured to sense user features and to send data related to the senseduser features to the processing system, the processing system beingconfigured to scan for user expressions as a function of the datarelated to the sensed user features.
 3. An aerial system, as set forthin claim 2, wherein the additional sensors are coupled to the body. 4.An aerial system, as set forth in claim 2, wherein the additionalsensors are located at a remote device, the remote device configured tosend the data related to the sensed user features to the aerial systemfor processing by the processing system.
 5. An aerial system, as setforth in claim 2, wherein the processing system utilizes facialrecognition and/or gesture recognition to identify user expressions as afunction of the data.
 6. An aerial system, as set forth in claim 5,wherein the processing system is configured to manage a release andhover process, the processing system being configured to control theaerial system to one of: (1) rotate and scan for the target person and(2) wait for the target person to enter a file of vision of the opticalsystem.
 7. An aerial system, as set forth in claim 6, wherein theprocessing system scans for user expressions after the target person hasbeen identified.
 8. An aerial system, as set forth in claim 7, whereinthe target person is identified by comparing data received from theadditional sensor(s) with pre-registered data.
 9. An aerial system, asset forth in claim 8, wherein the processing system in furtherconfigured to identify specific user expressions and to perform anassociated task.
 10. An aerial system, as set forth in claim 9, whereinthe user expressions are gestures, the processing system beingconfigured to identify one or more of the following gestures: thumb up,thumb down, open palm, fist, and victory gesture.
 11. An aerial system,as set forth in claim 1, wherein the processing system is configured toidentify and track movement of the open palm gesture relative to a faceof the target person.
 12. An aerial system, as set forth in claim 11,wherein the processing system is configured to move the aerial system asa function of the relative movement of the open palm gesture relative tothe face of the target person.
 13. An aerial system, as set forth inclaim 8, wherein the associated task includes one of the following: takea picture or snapshot, initiate shooting video, initiate auto-follow,starting auto-capturing routine, terminate routine, land, andauto-capture.
 14. An aerial system, as set forth in claim 2, wherein theadditional sensor(s) for sensing user features includes one or morecamera(s) of the optical system and/or a microphone, wherein the userexpressions are facial expressions and/or gestures and/or voicecommands.
 15. A method for controlling an aerial system including abody, a lift mechanism coupled to the body, an orientation sensorcoupled to the body for sensing an orientation of the body, an opticalsystem, and a processing system, the method comprising the processingsystem performing the steps of: detecting a release event in which theaerial system is released from being physically held by a user;operating the lift mechanism to enter a hover idle state in which theaerial system searches for a target person; recognizing the targetperson; tracking the target person and scanning for user expressions;identifying a hand of the target person; detecting a first predefineduser expression including an open palm gesture of the hand of the targetperson and responsively tracking a movement of the hand in the open palmgesture and executing a first corresponding event including operatingthe lift mechanism to fly the aerial system in a direction of thetracked movement of the hand in the open palm gesture while maintainingthe target person within a field of view of the optical system;detecting a second predefined user expression including a “V” signgesture by the hand of the target person and responsively execute acorresponding second event including operating the optical system totake a picture of the target person; entering the hover idle state uponcompleting the second corresponding event; detecting a standby eventincluding a change in an orientation vector of the body upon sensing achange in an orientation sensor signal received from the orientationsensor; and responsively operating the lift mechanism in a standby mode.16. A method, as set forth in claim 15, including the steps of: sensinguser features, using one or more additional sensors and sending datarelated to the sensed user features to the processing system, andscanning for user expressions as a function of the data related to thesensed user features.
 17. A method, as set forth in claim 16, whereinthe additional sensors are coupled to the body.
 18. A method, as setforth in claim 16, wherein the additional sensors are located at aremote device, the remote device configured to send the data related tothe sensed user features to the aerial system for processing by theprocessing system.
 19. A method, as set forth in claim 16, including thestep of utilizing, by the processing system, facial recognition and/orgesture recognition to identify user expressions as a function of thedata.
 20. A method, as set forth in claim 19, including the steps ofmanaging, by the processing system, release and hover process andcontrolling the aerial system to one of: (1) rotate and scan for thetarget person and (2) wait for the target person to enter a file ofvision of the optical system.
 21. A method, as set forth in claim 20,including the step of scanning, by the processing system, for userexpressions after the target person has been identified.
 22. A method,as set forth in claim 21, wherein the target person is identified bycomparing data received from the additional sensor(s) withpre-registered data.
 23. A method, as set forth in claim 22, includingthe step of identifying, by the processing system, specific userexpressions and to perform an associated task.
 24. A method, as setforth in claim 23, wherein the user expressions are gestures, theprocessing system being configured to identify one or more of thefollowing gestures: thumb up, thumb down, open palm, fist, and victorygesture.
 25. A method, as set forth in claim 15, including the steps ofidentifying and tracking movement of the open palm gesture relative to aface of the target person.
 26. A method, as set forth in claim 25,including the step of moving the aerial system as a function of therelative movement of the open palm gesture relative to the face of thetarget person.
 27. A method, as set forth in claim 23, wherein theassociated task includes one of the following: take a picture orsnapshot, initiate shooting video, initiate auto-follow, startingauto-capturing routine, terminate routine, land, and auto-capture.
 28. Amethod, as set forth in claim 16, wherein the additional sensor(s) forsensing user features includes one or more camera(s) of the opticalsystem and/or a microphone, wherein the user expressions are facialexpressions and/or gestures and/or voice commands.